{"action":"created","comment":{"url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/comments/30713673","id":30713673,"diff_hunk":"@@ -0,0 +1,41 @@\n+#!/bin/sh\n+\n+usage()\n+{\n+    echo \"Usage: $0 <region>\"\n+    echo \"where\"\n+    echo \"  <region> is one of ORD, DFW, IAD, LON, SYD, HKG\"\n+    exit 1\n+}\n+\n+if [ $# -lt 1 ]; then\n+    usage\n+fi\n+region=$1\n+\n+# get current time in format H(0..23)MM(00..59)\n+now=$(date +%k%M)\n+\n+# exit unless current time is in the right window\n+if [ $now -lt 530 -o $now -gt 630 ]; then","path":"usmu/src/main/resources/monitor-feeds_dump.sh","position":20,"original_position":20,"commit_id":"1fcc392b4c7a6a828b5b0898650cd33b70dc0957","original_commit_id":"1fcc392b4c7a6a828b5b0898650cd33b70dc0957","user":{"login":"usnavi","id":1426961,"avatar_url":"https://avatars.githubusercontent.com/u/1426961?v=3","gravatar_id":"","url":"https://api.github.com/users/usnavi","html_url":"https://github.com/usnavi","followers_url":"https://api.github.com/users/usnavi/followers","following_url":"https://api.github.com/users/usnavi/following{/other_user}","gists_url":"https://api.github.com/users/usnavi/gists{/gist_id}","starred_url":"https://api.github.com/users/usnavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/usnavi/subscriptions","organizations_url":"https://api.github.com/users/usnavi/orgs","repos_url":"https://api.github.com/users/usnavi/repos","events_url":"https://api.github.com/users/usnavi/events{/privacy}","received_events_url":"https://api.github.com/users/usnavi/received_events","type":"User","site_admin":false},"body":"My understanding is that we'd always check for a success file, but the date is determined by the current time.  E.g., before 6:30AM  5/20/2015, look for 5/19/2015 because today's job probably hasn't completed yet, and after 6:30AM, look for 5/20/2015\r\n\r\nIt seems like if the job runs long, we'd never see it turn from failed to success if we only check for success within a particular window. ","created_at":"2015-05-20T15:21:43Z","updated_at":"2015-05-20T15:21:43Z","html_url":"https://github.com/rackerlabs/cloudfeeds-nabu/pull/18#discussion_r30713673","pull_request_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/18","_links":{"self":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/comments/30713673"},"html":{"href":"https://github.com/rackerlabs/cloudfeeds-nabu/pull/18#discussion_r30713673"},"pull_request":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/18"}}},"pull_request":{"url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/18","id":35771717,"html_url":"https://github.com/rackerlabs/cloudfeeds-nabu/pull/18","diff_url":"https://github.com/rackerlabs/cloudfeeds-nabu/pull/18.diff","patch_url":"https://github.com/rackerlabs/cloudfeeds-nabu/pull/18.patch","issue_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/issues/18","number":18,"state":"open","locked":false,"title":"CF-836 Implement monitoring for Archiving apps","user":{"login":"VinnyQ","id":203223,"avatar_url":"https://avatars.githubusercontent.com/u/203223?v=3","gravatar_id":"","url":"https://api.github.com/users/VinnyQ","html_url":"https://github.com/VinnyQ","followers_url":"https://api.github.com/users/VinnyQ/followers","following_url":"https://api.github.com/users/VinnyQ/following{/other_user}","gists_url":"https://api.github.com/users/VinnyQ/gists{/gist_id}","starred_url":"https://api.github.com/users/VinnyQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VinnyQ/subscriptions","organizations_url":"https://api.github.com/users/VinnyQ/orgs","repos_url":"https://api.github.com/users/VinnyQ/repos","events_url":"https://api.github.com/users/VinnyQ/events{/privacy}","received_events_url":"https://api.github.com/users/VinnyQ/received_events","type":"User","site_admin":false},"body":"Add monitor shell scripts for hadoop success files for \r\n* feeds dump from ballista (monitor-feeds_dump.sh),\r\n* preference service dump from ballista (monitor-prefs_dump.sh), and \r\n* usmu processing run (monitor-usmu.sh)\r\n\r\nmonitor-feeds_dump.sh and monitor-usmu.sh requires a region (DFW, ORD, IAD, LON, HKG, SYD), while monitor-prefs_dump.sh does not.  All three will check for the success file of yesterday run, if the shell script is executed within a time window (currently set to between 5:30am and 6:30am UTC, but can be changed once we have the schedule finalized).\r\n\r\nThe monitor-prefs_dump.sh is written with the assumption that the success file will be stored in the hadoop fs path \"/user/cloudfeeds/prefs_dump/_$yesterday/_SUCCESS\" (note the underscore before $yesterday date value), which will be played in the near future.  \r\n\r\nSee Greg's comment in the Archiving Monitoring and Alerting wiki:\r\nhttps://one.rackspace.com/display/cloudfeeds/Archiving+Monitoring+and+Alerting?focusedCommentId=127133310#comment-127133310\r\n\r\nAlso see the *Success File Locations* section in the wiki to validate the path:\r\nhttps://one.rackspace.com/display/cloudfeeds/Archiving+Monitoring+and+Alerting?focusedCommentId=127133310#ArchivingMonitoringandAlerting-SuccessFileLocations","created_at":"2015-05-19T20:33:32Z","updated_at":"2015-05-20T15:21:43Z","closed_at":null,"merged_at":null,"merge_commit_sha":"89099c835e76072b4fe467406ad83eed981988e1","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/18/commits","review_comments_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/18/comments","review_comment_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/comments{/number}","comments_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/issues/18/comments","statuses_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/statuses/1fcc392b4c7a6a828b5b0898650cd33b70dc0957","head":{"label":"VinnyQ:CF-836_monitoring_for_archiving","ref":"CF-836_monitoring_for_archiving","sha":"1fcc392b4c7a6a828b5b0898650cd33b70dc0957","user":{"login":"VinnyQ","id":203223,"avatar_url":"https://avatars.githubusercontent.com/u/203223?v=3","gravatar_id":"","url":"https://api.github.com/users/VinnyQ","html_url":"https://github.com/VinnyQ","followers_url":"https://api.github.com/users/VinnyQ/followers","following_url":"https://api.github.com/users/VinnyQ/following{/other_user}","gists_url":"https://api.github.com/users/VinnyQ/gists{/gist_id}","starred_url":"https://api.github.com/users/VinnyQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VinnyQ/subscriptions","organizations_url":"https://api.github.com/users/VinnyQ/orgs","repos_url":"https://api.github.com/users/VinnyQ/repos","events_url":"https://api.github.com/users/VinnyQ/events{/privacy}","received_events_url":"https://api.github.com/users/VinnyQ/received_events","type":"User","site_admin":false},"repo":{"id":35685005,"name":"cloudfeeds-nabu","full_name":"VinnyQ/cloudfeeds-nabu","owner":{"login":"VinnyQ","id":203223,"avatar_url":"https://avatars.githubusercontent.com/u/203223?v=3","gravatar_id":"","url":"https://api.github.com/users/VinnyQ","html_url":"https://github.com/VinnyQ","followers_url":"https://api.github.com/users/VinnyQ/followers","following_url":"https://api.github.com/users/VinnyQ/following{/other_user}","gists_url":"https://api.github.com/users/VinnyQ/gists{/gist_id}","starred_url":"https://api.github.com/users/VinnyQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VinnyQ/subscriptions","organizations_url":"https://api.github.com/users/VinnyQ/orgs","repos_url":"https://api.github.com/users/VinnyQ/repos","events_url":"https://api.github.com/users/VinnyQ/events{/privacy}","received_events_url":"https://api.github.com/users/VinnyQ/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/VinnyQ/cloudfeeds-nabu","description":"Cloud Feeds Archiving solution","fork":true,"url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu","forks_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/forks","keys_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/keys{/key_id}","collaborators_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/teams","hooks_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/hooks","issue_events_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/issues/events{/number}","events_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/events","assignees_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/assignees{/user}","branches_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/branches{/branch}","tags_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/tags","blobs_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/git/refs{/sha}","trees_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/git/trees{/sha}","statuses_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/statuses/{sha}","languages_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/languages","stargazers_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/stargazers","contributors_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/contributors","subscribers_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/subscribers","subscription_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/subscription","commits_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/commits{/sha}","git_commits_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/git/commits{/sha}","comments_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/comments{/number}","issue_comment_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/issues/comments{/number}","contents_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/contents/{+path}","compare_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/compare/{base}...{head}","merges_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/merges","archive_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/downloads","issues_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/issues{/number}","pulls_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/pulls{/number}","milestones_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/milestones{/number}","notifications_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/labels{/name}","releases_url":"https://api.github.com/repos/VinnyQ/cloudfeeds-nabu/releases{/id}","created_at":"2015-05-15T16:38:18Z","updated_at":"2015-05-15T16:38:18Z","pushed_at":"2015-05-19T20:25:59Z","git_url":"git://github.com/VinnyQ/cloudfeeds-nabu.git","ssh_url":"git@github.com:VinnyQ/cloudfeeds-nabu.git","clone_url":"https://github.com/VinnyQ/cloudfeeds-nabu.git","svn_url":"https://github.com/VinnyQ/cloudfeeds-nabu","homepage":null,"size":186,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"rackerlabs:master","ref":"master","sha":"31fb6270e01a8a27089a9c84f5b7e27fe906c41b","user":{"login":"rackerlabs","id":84293,"avatar_url":"https://avatars.githubusercontent.com/u/84293?v=3","gravatar_id":"","url":"https://api.github.com/users/rackerlabs","html_url":"https://github.com/rackerlabs","followers_url":"https://api.github.com/users/rackerlabs/followers","following_url":"https://api.github.com/users/rackerlabs/following{/other_user}","gists_url":"https://api.github.com/users/rackerlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/rackerlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rackerlabs/subscriptions","organizations_url":"https://api.github.com/users/rackerlabs/orgs","repos_url":"https://api.github.com/users/rackerlabs/repos","events_url":"https://api.github.com/users/rackerlabs/events{/privacy}","received_events_url":"https://api.github.com/users/rackerlabs/received_events","type":"Organization","site_admin":false},"repo":{"id":30371667,"name":"cloudfeeds-nabu","full_name":"rackerlabs/cloudfeeds-nabu","owner":{"login":"rackerlabs","id":84293,"avatar_url":"https://avatars.githubusercontent.com/u/84293?v=3","gravatar_id":"","url":"https://api.github.com/users/rackerlabs","html_url":"https://github.com/rackerlabs","followers_url":"https://api.github.com/users/rackerlabs/followers","following_url":"https://api.github.com/users/rackerlabs/following{/other_user}","gists_url":"https://api.github.com/users/rackerlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/rackerlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rackerlabs/subscriptions","organizations_url":"https://api.github.com/users/rackerlabs/orgs","repos_url":"https://api.github.com/users/rackerlabs/repos","events_url":"https://api.github.com/users/rackerlabs/events{/privacy}","received_events_url":"https://api.github.com/users/rackerlabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/rackerlabs/cloudfeeds-nabu","description":"Cloud Feeds Archiving solution","fork":false,"url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu","forks_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/forks","keys_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/teams","hooks_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/hooks","issue_events_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/issues/events{/number}","events_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/events","assignees_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/assignees{/user}","branches_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/branches{/branch}","tags_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/tags","blobs_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/git/refs{/sha}","trees_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/statuses/{sha}","languages_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/languages","stargazers_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/stargazers","contributors_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/contributors","subscribers_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/subscribers","subscription_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/subscription","commits_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/commits{/sha}","git_commits_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/git/commits{/sha}","comments_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/comments{/number}","issue_comment_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/issues/comments{/number}","contents_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/contents/{+path}","compare_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/merges","archive_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/downloads","issues_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/issues{/number}","pulls_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls{/number}","milestones_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/milestones{/number}","notifications_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/labels{/name}","releases_url":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/releases{/id}","created_at":"2015-02-05T18:40:02Z","updated_at":"2015-04-23T14:34:48Z","pushed_at":"2015-05-19T20:33:33Z","git_url":"git://github.com/rackerlabs/cloudfeeds-nabu.git","ssh_url":"git@github.com:rackerlabs/cloudfeeds-nabu.git","clone_url":"https://github.com/rackerlabs/cloudfeeds-nabu.git","svn_url":"https://github.com/rackerlabs/cloudfeeds-nabu","homepage":null,"size":610,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":5,"mirror_url":null,"open_issues_count":1,"forks":5,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/18"},"html":{"href":"https://github.com/rackerlabs/cloudfeeds-nabu/pull/18"},"issue":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/issues/18"},"comments":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/issues/18/comments"},"review_comments":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/18/comments"},"review_comment":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/pulls/18/commits"},"statuses":{"href":"https://api.github.com/repos/rackerlabs/cloudfeeds-nabu/statuses/1fcc392b4c7a6a828b5b0898650cd33b70dc0957"}}}}