{"action":"created","comment":{"url":"https://api.github.com/repos/fedora-infra/statscache/pulls/comments/37334546","id":37334546,"diff_hunk":"@@ -12,95 +11,35 @@\n log = logging.getLogger(\"fedmsg\")\n \n \n-class StatsProducerBase(moksha.hub.api.PollingProducer):\n+class StatsProducer(moksha.hub.api.PollingProducer):\n     \"\"\"\n     An abstract base class for dynamically generated producers. Subclasses need","path":"statscache/producer.py","position":15,"original_position":15,"commit_id":"1f9305346f80141ffa5f62d56281d20f0c0fed68","original_commit_id":"1f9305346f80141ffa5f62d56281d20f0c0fed68","user":{"login":"yazman","id":4121048,"avatar_url":"https://avatars.githubusercontent.com/u/4121048?v=3","gravatar_id":"","url":"https://api.github.com/users/yazman","html_url":"https://github.com/yazman","followers_url":"https://api.github.com/users/yazman/followers","following_url":"https://api.github.com/users/yazman/following{/other_user}","gists_url":"https://api.github.com/users/yazman/gists{/gist_id}","starred_url":"https://api.github.com/users/yazman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yazman/subscriptions","organizations_url":"https://api.github.com/users/yazman/orgs","repos_url":"https://api.github.com/users/yazman/repos","events_url":"https://api.github.com/users/yazman/events{/privacy}","received_events_url":"https://api.github.com/users/yazman/received_events","type":"User","site_admin":false},"body":"Oh no you're right; that definitely is out-of-date. Nice catch!","created_at":"2015-08-18T18:24:55Z","updated_at":"2015-08-18T18:24:55Z","html_url":"https://github.com/fedora-infra/statscache/pull/28#discussion_r37334546","pull_request_url":"https://api.github.com/repos/fedora-infra/statscache/pulls/28","_links":{"self":{"href":"https://api.github.com/repos/fedora-infra/statscache/pulls/comments/37334546"},"html":{"href":"https://github.com/fedora-infra/statscache/pull/28#discussion_r37334546"},"pull_request":{"href":"https://api.github.com/repos/fedora-infra/statscache/pulls/28"}}},"pull_request":{"url":"https://api.github.com/repos/fedora-infra/statscache/pulls/28","id":42648842,"html_url":"https://github.com/fedora-infra/statscache/pull/28","diff_url":"https://github.com/fedora-infra/statscache/pull/28.diff","patch_url":"https://github.com/fedora-infra/statscache/pull/28.patch","issue_url":"https://api.github.com/repos/fedora-infra/statscache/issues/28","number":28,"state":"open","locked":false,"title":"Real-time message processing and synchronized database updates","user":{"login":"yazman","id":4121048,"avatar_url":"https://avatars.githubusercontent.com/u/4121048?v=3","gravatar_id":"","url":"https://api.github.com/users/yazman","html_url":"https://github.com/yazman","followers_url":"https://api.github.com/users/yazman/followers","following_url":"https://api.github.com/users/yazman/following{/other_user}","gists_url":"https://api.github.com/users/yazman/gists{/gist_id}","starred_url":"https://api.github.com/users/yazman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yazman/subscriptions","organizations_url":"https://api.github.com/users/yazman/orgs","repos_url":"https://api.github.com/users/yazman/repos","events_url":"https://api.github.com/users/yazman/events{/privacy}","received_events_url":"https://api.github.com/users/yazman/received_events","type":"User","site_admin":false},"body":"I've written a working implementation of my second proposed re-architecturing solution, described in my issue [here](https://github.com/fedora-infra/statscache/issues/23#issuecomment-126097527). During normal operation, this is not that big of a change. When a message is received, a deep copy is directly handed to each plugin by the fedmsg consumer for processing, and each plugin stores whatever the results are of this processing internally. For example, volume plugins do something along the lines of ``self._volumes[(msg.timestamp, key(msg))]  += 1`` where ``self._volumes`` is a ``collections.defaultdict``. On a configurable frequency, a polling producer instructs each plugin to update its database model with its stored results.\r\n\r\nAt startup, things do change quite a bit. For one, there is no longer a need for such a complicated plugin/producer initialization relationship (where the polling producers needed to be generated after plugins were loaded but before they were instantiated). Additionally, the plugin ``initialize()`` method, used to process backlogged messages, is removed in favor of doing this centrally in the fedmsg consumer initialization. This is good for encapsulation, as all operations related to message retrieval, receipt, and distribution happen in the same class. Since patchy data can't give us all that much insight, I've also added a configuration value for the \"epoch\" of statistics. This is the date and time as of which statscache will have uninterruptedly processed all fedmsg traffic. However, some plugins (the releng one in particular comes to mind) really do only care about somewhat recent messages. To support this, plugins are able to modify this continuity, basically by lying about how up-to-date they are (*i.e.*, reporting that they are up-to-date as of ``max(datetime.now() - self.backlog_delta, latest_row.timestamp)``). Of course, it must then be noted that historical data in such models will be spotty or possibly even nonexistent.\r\n\r\nNote: Due to the fact that consensus was never reached in the original issue, please feel free to consider this an RFC or WIP if there are objections.","created_at":"2015-08-18T00:11:22Z","updated_at":"2015-08-18T18:24:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6b68a7d9da267ae28b8069c3865ad80dc0b456ad","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/fedora-infra/statscache/pulls/28/commits","review_comments_url":"https://api.github.com/repos/fedora-infra/statscache/pulls/28/comments","review_comment_url":"https://api.github.com/repos/fedora-infra/statscache/pulls/comments{/number}","comments_url":"https://api.github.com/repos/fedora-infra/statscache/issues/28/comments","statuses_url":"https://api.github.com/repos/fedora-infra/statscache/statuses/1f9305346f80141ffa5f62d56281d20f0c0fed68","head":{"label":"yazman:realtime","ref":"realtime","sha":"1f9305346f80141ffa5f62d56281d20f0c0fed68","user":{"login":"yazman","id":4121048,"avatar_url":"https://avatars.githubusercontent.com/u/4121048?v=3","gravatar_id":"","url":"https://api.github.com/users/yazman","html_url":"https://github.com/yazman","followers_url":"https://api.github.com/users/yazman/followers","following_url":"https://api.github.com/users/yazman/following{/other_user}","gists_url":"https://api.github.com/users/yazman/gists{/gist_id}","starred_url":"https://api.github.com/users/yazman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yazman/subscriptions","organizations_url":"https://api.github.com/users/yazman/orgs","repos_url":"https://api.github.com/users/yazman/repos","events_url":"https://api.github.com/users/yazman/events{/privacy}","received_events_url":"https://api.github.com/users/yazman/received_events","type":"User","site_admin":false},"repo":{"id":37870874,"name":"statscache","full_name":"yazman/statscache","owner":{"login":"yazman","id":4121048,"avatar_url":"https://avatars.githubusercontent.com/u/4121048?v=3","gravatar_id":"","url":"https://api.github.com/users/yazman","html_url":"https://github.com/yazman","followers_url":"https://api.github.com/users/yazman/followers","following_url":"https://api.github.com/users/yazman/following{/other_user}","gists_url":"https://api.github.com/users/yazman/gists{/gist_id}","starred_url":"https://api.github.com/users/yazman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yazman/subscriptions","organizations_url":"https://api.github.com/users/yazman/orgs","repos_url":"https://api.github.com/users/yazman/repos","events_url":"https://api.github.com/users/yazman/events{/privacy}","received_events_url":"https://api.github.com/users/yazman/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/yazman/statscache","description":"A daemon to build and keep fedmsg statistics","fork":true,"url":"https://api.github.com/repos/yazman/statscache","forks_url":"https://api.github.com/repos/yazman/statscache/forks","keys_url":"https://api.github.com/repos/yazman/statscache/keys{/key_id}","collaborators_url":"https://api.github.com/repos/yazman/statscache/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/yazman/statscache/teams","hooks_url":"https://api.github.com/repos/yazman/statscache/hooks","issue_events_url":"https://api.github.com/repos/yazman/statscache/issues/events{/number}","events_url":"https://api.github.com/repos/yazman/statscache/events","assignees_url":"https://api.github.com/repos/yazman/statscache/assignees{/user}","branches_url":"https://api.github.com/repos/yazman/statscache/branches{/branch}","tags_url":"https://api.github.com/repos/yazman/statscache/tags","blobs_url":"https://api.github.com/repos/yazman/statscache/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/yazman/statscache/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/yazman/statscache/git/refs{/sha}","trees_url":"https://api.github.com/repos/yazman/statscache/git/trees{/sha}","statuses_url":"https://api.github.com/repos/yazman/statscache/statuses/{sha}","languages_url":"https://api.github.com/repos/yazman/statscache/languages","stargazers_url":"https://api.github.com/repos/yazman/statscache/stargazers","contributors_url":"https://api.github.com/repos/yazman/statscache/contributors","subscribers_url":"https://api.github.com/repos/yazman/statscache/subscribers","subscription_url":"https://api.github.com/repos/yazman/statscache/subscription","commits_url":"https://api.github.com/repos/yazman/statscache/commits{/sha}","git_commits_url":"https://api.github.com/repos/yazman/statscache/git/commits{/sha}","comments_url":"https://api.github.com/repos/yazman/statscache/comments{/number}","issue_comment_url":"https://api.github.com/repos/yazman/statscache/issues/comments{/number}","contents_url":"https://api.github.com/repos/yazman/statscache/contents/{+path}","compare_url":"https://api.github.com/repos/yazman/statscache/compare/{base}...{head}","merges_url":"https://api.github.com/repos/yazman/statscache/merges","archive_url":"https://api.github.com/repos/yazman/statscache/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/yazman/statscache/downloads","issues_url":"https://api.github.com/repos/yazman/statscache/issues{/number}","pulls_url":"https://api.github.com/repos/yazman/statscache/pulls{/number}","milestones_url":"https://api.github.com/repos/yazman/statscache/milestones{/number}","notifications_url":"https://api.github.com/repos/yazman/statscache/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/yazman/statscache/labels{/name}","releases_url":"https://api.github.com/repos/yazman/statscache/releases{/id}","created_at":"2015-06-22T17:48:00Z","updated_at":"2015-06-22T17:48:00Z","pushed_at":"2015-08-18T00:44:19Z","git_url":"git://github.com/yazman/statscache.git","ssh_url":"git@github.com:yazman/statscache.git","clone_url":"https://github.com/yazman/statscache.git","svn_url":"https://github.com/yazman/statscache","homepage":"","size":1370,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"fedora-infra:develop","ref":"develop","sha":"d4028d59c52c2d402d2b79e1a6bf7a42a36e800b","user":{"login":"fedora-infra","id":3316637,"avatar_url":"https://avatars.githubusercontent.com/u/3316637?v=3","gravatar_id":"","url":"https://api.github.com/users/fedora-infra","html_url":"https://github.com/fedora-infra","followers_url":"https://api.github.com/users/fedora-infra/followers","following_url":"https://api.github.com/users/fedora-infra/following{/other_user}","gists_url":"https://api.github.com/users/fedora-infra/gists{/gist_id}","starred_url":"https://api.github.com/users/fedora-infra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fedora-infra/subscriptions","organizations_url":"https://api.github.com/users/fedora-infra/orgs","repos_url":"https://api.github.com/users/fedora-infra/repos","events_url":"https://api.github.com/users/fedora-infra/events{/privacy}","received_events_url":"https://api.github.com/users/fedora-infra/received_events","type":"Organization","site_admin":false},"repo":{"id":33745687,"name":"statscache","full_name":"fedora-infra/statscache","owner":{"login":"fedora-infra","id":3316637,"avatar_url":"https://avatars.githubusercontent.com/u/3316637?v=3","gravatar_id":"","url":"https://api.github.com/users/fedora-infra","html_url":"https://github.com/fedora-infra","followers_url":"https://api.github.com/users/fedora-infra/followers","following_url":"https://api.github.com/users/fedora-infra/following{/other_user}","gists_url":"https://api.github.com/users/fedora-infra/gists{/gist_id}","starred_url":"https://api.github.com/users/fedora-infra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fedora-infra/subscriptions","organizations_url":"https://api.github.com/users/fedora-infra/orgs","repos_url":"https://api.github.com/users/fedora-infra/repos","events_url":"https://api.github.com/users/fedora-infra/events{/privacy}","received_events_url":"https://api.github.com/users/fedora-infra/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/fedora-infra/statscache","description":"A daemon to build and keep fedmsg statistics","fork":false,"url":"https://api.github.com/repos/fedora-infra/statscache","forks_url":"https://api.github.com/repos/fedora-infra/statscache/forks","keys_url":"https://api.github.com/repos/fedora-infra/statscache/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fedora-infra/statscache/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fedora-infra/statscache/teams","hooks_url":"https://api.github.com/repos/fedora-infra/statscache/hooks","issue_events_url":"https://api.github.com/repos/fedora-infra/statscache/issues/events{/number}","events_url":"https://api.github.com/repos/fedora-infra/statscache/events","assignees_url":"https://api.github.com/repos/fedora-infra/statscache/assignees{/user}","branches_url":"https://api.github.com/repos/fedora-infra/statscache/branches{/branch}","tags_url":"https://api.github.com/repos/fedora-infra/statscache/tags","blobs_url":"https://api.github.com/repos/fedora-infra/statscache/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fedora-infra/statscache/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fedora-infra/statscache/git/refs{/sha}","trees_url":"https://api.github.com/repos/fedora-infra/statscache/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fedora-infra/statscache/statuses/{sha}","languages_url":"https://api.github.com/repos/fedora-infra/statscache/languages","stargazers_url":"https://api.github.com/repos/fedora-infra/statscache/stargazers","contributors_url":"https://api.github.com/repos/fedora-infra/statscache/contributors","subscribers_url":"https://api.github.com/repos/fedora-infra/statscache/subscribers","subscription_url":"https://api.github.com/repos/fedora-infra/statscache/subscription","commits_url":"https://api.github.com/repos/fedora-infra/statscache/commits{/sha}","git_commits_url":"https://api.github.com/repos/fedora-infra/statscache/git/commits{/sha}","comments_url":"https://api.github.com/repos/fedora-infra/statscache/comments{/number}","issue_comment_url":"https://api.github.com/repos/fedora-infra/statscache/issues/comments{/number}","contents_url":"https://api.github.com/repos/fedora-infra/statscache/contents/{+path}","compare_url":"https://api.github.com/repos/fedora-infra/statscache/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fedora-infra/statscache/merges","archive_url":"https://api.github.com/repos/fedora-infra/statscache/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fedora-infra/statscache/downloads","issues_url":"https://api.github.com/repos/fedora-infra/statscache/issues{/number}","pulls_url":"https://api.github.com/repos/fedora-infra/statscache/pulls{/number}","milestones_url":"https://api.github.com/repos/fedora-infra/statscache/milestones{/number}","notifications_url":"https://api.github.com/repos/fedora-infra/statscache/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fedora-infra/statscache/labels{/name}","releases_url":"https://api.github.com/repos/fedora-infra/statscache/releases{/id}","created_at":"2015-04-10T19:03:38Z","updated_at":"2015-06-28T00:07:47Z","pushed_at":"2015-08-18T00:44:19Z","git_url":"git://github.com/fedora-infra/statscache.git","ssh_url":"git@github.com:fedora-infra/statscache.git","clone_url":"https://github.com/fedora-infra/statscache.git","svn_url":"https://github.com/fedora-infra/statscache","homepage":"","size":411,"stargazers_count":3,"watchers_count":3,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":7,"forks":2,"open_issues":7,"watchers":3,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/fedora-infra/statscache/pulls/28"},"html":{"href":"https://github.com/fedora-infra/statscache/pull/28"},"issue":{"href":"https://api.github.com/repos/fedora-infra/statscache/issues/28"},"comments":{"href":"https://api.github.com/repos/fedora-infra/statscache/issues/28/comments"},"review_comments":{"href":"https://api.github.com/repos/fedora-infra/statscache/pulls/28/comments"},"review_comment":{"href":"https://api.github.com/repos/fedora-infra/statscache/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/fedora-infra/statscache/pulls/28/commits"},"statuses":{"href":"https://api.github.com/repos/fedora-infra/statscache/statuses/1f9305346f80141ffa5f62d56281d20f0c0fed68"}}}}