{"action":"created","comment":{"url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/comments/46685269","id":46685269,"diff_hunk":"@@ -0,0 +1,36 @@\n+# coding: utf-8\n+#\n+# Count hits to each url in the log files\n+#\n+# Ensures that there are counts in the `<processed_dir>/raw_counts/` tree for\n+# all log files which exist.\n+#\n+# If log files are removed, the counts are retained, so that we can archive the\n+# logs but still do historical analysis of trends.\n+\n+require_relative 'config_logging'\n+require_relative 'log_counter'\n+\n+class CountCdnLogs\n+  def initialize(log_dir, processed_dir)\n+    @log_dir = log_dir\n+    @processed_dir = processed_dir\n+    @counts_dir = \"#{processed_dir}/raw_counts\"\n+  end\n+\n+  def update\n+    files = completed_log_files","path":"lib/count_cdn_logs.rb","position":22,"original_position":22,"commit_id":"da6d563a365b8eb7221e6e9dc151a93d3c95a0f6","original_commit_id":"da6d563a365b8eb7221e6e9dc151a93d3c95a0f6","user":{"login":"rboulton","id":73564,"avatar_url":"https://avatars.githubusercontent.com/u/73564?v=3","gravatar_id":"","url":"https://api.github.com/users/rboulton","html_url":"https://github.com/rboulton","followers_url":"https://api.github.com/users/rboulton/followers","following_url":"https://api.github.com/users/rboulton/following{/other_user}","gists_url":"https://api.github.com/users/rboulton/gists{/gist_id}","starred_url":"https://api.github.com/users/rboulton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rboulton/subscriptions","organizations_url":"https://api.github.com/users/rboulton/orgs","repos_url":"https://api.github.com/users/rboulton/repos","events_url":"https://api.github.com/users/rboulton/events{/privacy}","received_events_url":"https://api.github.com/users/rboulton/received_events","type":"User","site_admin":false},"body":"I was actually wanting to avoid evaluating completed_log_files twice, in case the result was changed (eg, by an accidentally concurrently running program). But I failed to actually do that.\r\n\r\nInstead, I'm changing this to memoize the result of completed_log_files, as well as calling it directly as suggested.","created_at":"2015-12-04T14:15:13Z","updated_at":"2015-12-04T14:15:13Z","html_url":"https://github.com/alphagov/govuk-cdn-logs-monitor/pull/3#discussion_r46685269","pull_request_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/3","_links":{"self":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/comments/46685269"},"html":{"href":"https://github.com/alphagov/govuk-cdn-logs-monitor/pull/3#discussion_r46685269"},"pull_request":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/3"}}},"pull_request":{"url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/3","id":52555583,"html_url":"https://github.com/alphagov/govuk-cdn-logs-monitor/pull/3","diff_url":"https://github.com/alphagov/govuk-cdn-logs-monitor/pull/3.diff","patch_url":"https://github.com/alphagov/govuk-cdn-logs-monitor/pull/3.patch","issue_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues/3","number":3,"state":"open","locked":false,"title":"Rewrite - a new approach","user":{"login":"rboulton","id":73564,"avatar_url":"https://avatars.githubusercontent.com/u/73564?v=3","gravatar_id":"","url":"https://api.github.com/users/rboulton","html_url":"https://github.com/rboulton","followers_url":"https://api.github.com/users/rboulton/followers","following_url":"https://api.github.com/users/rboulton/following{/other_user}","gists_url":"https://api.github.com/users/rboulton/gists{/gist_id}","starred_url":"https://api.github.com/users/rboulton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rboulton/subscriptions","organizations_url":"https://api.github.com/users/rboulton/orgs","repos_url":"https://api.github.com/users/rboulton/repos","events_url":"https://api.github.com/users/rboulton/events{/privacy}","received_events_url":"https://api.github.com/users/rboulton/received_events","type":"User","site_admin":false},"body":"This is an attempt to address several problems with the existing mix of shell and ruby scripts. Firstly, the mix of shell and ruby is a bit hard to follow.\r\n\r\nSecondly, the log files aren't actually for a single day, and sometimes we have several days logs in a single file (eg, because logrotate breaks for a bit), so we need to parse the timestamps in the file - which includes being able to parse timezone information, and then merge the resulting counts from different files to get the correct counts for a single day.\r\n\r\nThirdly, we want to be able to ensure that we have accurate counts for the log files which exist, even if they get modified (or changed to a compressed form) - so this design moves to having a single script for counting all the log files which are present, which will skip log files which have already been counted: it ensures that a log file which is later modified is re-counted by tracking the file size when the log was counted.  This script will be run nightly to update the derived counts.\r\n\r\nFinally, and perhaps most importantly, we may not be able to keep the GOV.UK logs indefinitely, due to data size and potential privacy implications, but a digest of just the counts of accesses to URLs in a time period will take up much less space, have fewer privacy risks, and be almost as useful for data analysis. We expect in future to want to do more than just look for successful requests in the logs, so this commit takes the approach of first running a pure count of the combination of hour of day, HTTP method, path, and resulting status code.\r\n\r\nLater stages of processing are then done to using the counts as input, producing derived information such as lists of successfully accessed pages per day.  This processing can be adjusted as needed, without the need to refer back to the original source log files (which can then be deleted if needed without affecting the future options for analysis).","created_at":"2015-12-03T16:16:20Z","updated_at":"2015-12-04T14:15:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"fad1f2032794942e4809fed115c64e877fa34b31","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/3/commits","review_comments_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/3/comments","review_comment_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/comments{/number}","comments_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues/3/comments","statuses_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/statuses/da6d563a365b8eb7221e6e9dc151a93d3c95a0f6","head":{"label":"alphagov:rebuild","ref":"rebuild","sha":"da6d563a365b8eb7221e6e9dc151a93d3c95a0f6","user":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":42452289,"name":"govuk-cdn-logs-monitor","full_name":"alphagov/govuk-cdn-logs-monitor","owner":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/alphagov/govuk-cdn-logs-monitor","description":"Monitor the CDN logs for GOV.UK to find problems with the site","fork":false,"url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor","forks_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/forks","keys_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/teams","hooks_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/hooks","issue_events_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/events","assignees_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/tags","blobs_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/languages","stargazers_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/stargazers","contributors_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/contributors","subscribers_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/subscribers","subscription_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/subscription","commits_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/merges","archive_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/downloads","issues_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/releases{/id}","created_at":"2015-09-14T14:04:35Z","updated_at":"2015-10-07T11:53:27Z","pushed_at":"2015-12-03T16:32:02Z","git_url":"git://github.com/alphagov/govuk-cdn-logs-monitor.git","ssh_url":"git@github.com:alphagov/govuk-cdn-logs-monitor.git","clone_url":"https://github.com/alphagov/govuk-cdn-logs-monitor.git","svn_url":"https://github.com/alphagov/govuk-cdn-logs-monitor","homepage":null,"size":365,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"alphagov:master","ref":"master","sha":"df7738e859d738d7f6c2d48cf8c5c7b2c13b5417","user":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":42452289,"name":"govuk-cdn-logs-monitor","full_name":"alphagov/govuk-cdn-logs-monitor","owner":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/alphagov/govuk-cdn-logs-monitor","description":"Monitor the CDN logs for GOV.UK to find problems with the site","fork":false,"url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor","forks_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/forks","keys_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/teams","hooks_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/hooks","issue_events_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/events","assignees_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/tags","blobs_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/languages","stargazers_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/stargazers","contributors_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/contributors","subscribers_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/subscribers","subscription_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/subscription","commits_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/merges","archive_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/downloads","issues_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/releases{/id}","created_at":"2015-09-14T14:04:35Z","updated_at":"2015-10-07T11:53:27Z","pushed_at":"2015-12-03T16:32:02Z","git_url":"git://github.com/alphagov/govuk-cdn-logs-monitor.git","ssh_url":"git@github.com:alphagov/govuk-cdn-logs-monitor.git","clone_url":"https://github.com/alphagov/govuk-cdn-logs-monitor.git","svn_url":"https://github.com/alphagov/govuk-cdn-logs-monitor","homepage":null,"size":365,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/3"},"html":{"href":"https://github.com/alphagov/govuk-cdn-logs-monitor/pull/3"},"issue":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues/3"},"comments":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/issues/3/comments"},"review_comments":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/3/comments"},"review_comment":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/pulls/3/commits"},"statuses":{"href":"https://api.github.com/repos/alphagov/govuk-cdn-logs-monitor/statuses/da6d563a365b8eb7221e6e9dc151a93d3c95a0f6"}}}}