{"action":"created","comment":{"url":"https://api.github.com/repos/buildbot/buildbot/pulls/comments/34292648","id":34292648,"diff_hunk":"@@ -0,0 +1,70 @@\n+# This file is part of Buildbot.  Buildbot is free software: you can\n+# redistribute it and/or modify it under the terms of the GNU General Public\n+# License as published by the Free Software Foundation, version 2.\n+#\n+# This program is distributed in the hope that it will be useful, but WITHOUT\n+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS\n+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more\n+# details.\n+#\n+# You should have received a copy of the GNU General Public License along with\n+# this program; if not, write to the Free Software Foundation, Inc., 51\n+# Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.\n+#\n+# Copyright Buildbot Team Members\n+\n+from twisted.internet import defer\n+from twisted.python import log\n+\n+from buildbot import config\n+\n+try:\n+    from influxdb import InfluxDBClient\n+except ImportError:\n+    InfluxDBClient = None\n+\n+\n+class StatsStorageBase(object):\n+    \"\"\"\n+    Base class for sub service responsible for passing on stats data to\n+    a storage backend\n+    \"\"\"\n+    def postStatsValue(self, name, value, series_name, context={}):\n+        return defer.succeed(None)\n+\n+\n+class InfluxStorageService(StatsStorageBase):\n+    \"\"\"\n+    Delegates data to InfluxDB\n+    \"\"\"\n+    def __init__(self, url, port, user, password, db, captures,\n+                 name=\"InfluxStorageService\"):\n+        if not InfluxDBClient:\n+            config.error(\"Python client for InfluxDB not installed.\")\n+        self.url = url","path":"master/buildbot/statistics/storage_backends.py","position":44,"original_position":44,"commit_id":"8ba9ac5f74dbda6ad2fac00db1d47f358f69c69d","original_commit_id":"8ba9ac5f74dbda6ad2fac00db1d47f358f69c69d","user":{"login":"prasoon2211","id":2565248,"avatar_url":"https://avatars.githubusercontent.com/u/2565248?v=3","gravatar_id":"","url":"https://api.github.com/users/prasoon2211","html_url":"https://github.com/prasoon2211","followers_url":"https://api.github.com/users/prasoon2211/followers","following_url":"https://api.github.com/users/prasoon2211/following{/other_user}","gists_url":"https://api.github.com/users/prasoon2211/gists{/gist_id}","starred_url":"https://api.github.com/users/prasoon2211/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prasoon2211/subscriptions","organizations_url":"https://api.github.com/users/prasoon2211/orgs","repos_url":"https://api.github.com/users/prasoon2211/repos","events_url":"https://api.github.com/users/prasoon2211/events{/privacy}","received_events_url":"https://api.github.com/users/prasoon2211/received_events","type":"User","site_admin":false},"body":"No. I'll prepend them with a `_`.","created_at":"2015-07-09T19:19:49Z","updated_at":"2015-07-09T19:19:49Z","html_url":"https://github.com/buildbot/buildbot/pull/1725#discussion_r34292648","pull_request_url":"https://api.github.com/repos/buildbot/buildbot/pulls/1725","_links":{"self":{"href":"https://api.github.com/repos/buildbot/buildbot/pulls/comments/34292648"},"html":{"href":"https://github.com/buildbot/buildbot/pull/1725#discussion_r34292648"},"pull_request":{"href":"https://api.github.com/repos/buildbot/buildbot/pulls/1725"}}},"pull_request":{"url":"https://api.github.com/repos/buildbot/buildbot/pulls/1725","id":37713959,"html_url":"https://github.com/buildbot/buildbot/pull/1725","diff_url":"https://github.com/buildbot/buildbot/pull/1725.diff","patch_url":"https://github.com/buildbot/buildbot/pull/1725.patch","issue_url":"https://api.github.com/repos/buildbot/buildbot/issues/1725","number":1725,"state":"open","locked":false,"title":"Statistics service for Buildbot","user":{"login":"prasoon2211","id":2565248,"avatar_url":"https://avatars.githubusercontent.com/u/2565248?v=3","gravatar_id":"","url":"https://api.github.com/users/prasoon2211","html_url":"https://github.com/prasoon2211","followers_url":"https://api.github.com/users/prasoon2211/followers","following_url":"https://api.github.com/users/prasoon2211/following{/other_user}","gists_url":"https://api.github.com/users/prasoon2211/gists{/gist_id}","starred_url":"https://api.github.com/users/prasoon2211/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prasoon2211/subscriptions","organizations_url":"https://api.github.com/users/prasoon2211/orgs","repos_url":"https://api.github.com/users/prasoon2211/repos","events_url":"https://api.github.com/users/prasoon2211/events{/privacy}","received_events_url":"https://api.github.com/users/prasoon2211/received_events","type":"User","site_admin":false},"body":"This PR adds support for statistics service. The aim of this project is to provide support for storing and visualizing data generated by builds gathered from within Buildbot (as graphs and charts). See: https://gist.github.com/prasoon2211/cbe93436acc436be61c9 for introduction.\r\n\r\nAs a first step, this PR adds support for storage of build properties in the InfluxDB storage database.\r\n\r\nHere's a brief outline of the the working:\r\n\r\n* The user initializes storage backends in the config file:\r\n```\r\ncaptures = [stats.CaptureProperty('runtests', 'tree-size-KiB')]\r\nc['statsServices'] = [stats.InfluxStorageService('localhost', 8086, 'root', 'root', 'test', captures)]\r\n```\r\n* `StatsService` gets started as a twisted service when Buildbot master is started.\r\n* A build is started (scheduled or forced).\r\n* When the build ends, all the build properties are sent to the running `StatsService`.\r\n* `StatsService.postProperties` then filters out the properties as defined in the config file and sends these properties to the proper storage backend (InfluxDB, in this case). For example, the above configuration will store the value of `tree-size-KiB` for the builder `runtests` in the storage backend (in this case the InfluxStorageService).\r\n* `InfluxStorageService`, which is a Python client for the InfluxDB database, is then responsible for sending this data to the actual running InfluxDB instance on the user's machine. The data is stored as a time-series in InfluxDB. This data is then accessible via the InfluxDB user frontend (that runs in a browser), via an HTTP API to InfluxDB or by using any number of language-specific clients for InfluxDB.\r\n\r\nWe (me and the mentors @Stibbons and @benallard) plan for the user to be able to use several storage backends. So, `c['statsServices']` will be a list of all the storage backends. Currently, we only support InfluxDB but we'll add support for other storage backends (e.g. Elasticsearch) later on. Even now, with only InfluxDB support, `c['statsServices']` can contain several `InfluxStorageServices`.\r\n\r\nDivision of work:\r\n\r\nThe code is divided into three logically separate commits:\r\n* The first commit adds the actual stats service ( + tests + docs).\r\n* The second commit adds support for storage backends - InfluxStorageService, in this case ( + tests + docs).\r\n* The third commit adds support for the capture module - it's very small right now but we plan to add all the capturing support to this (e.g. capturing build/start duration, start, end time etc.). This commit also contains the user documentation (the Buildbot manual).\r\n\r\nAt first, we wanted to make three separate PRs, one for each of the commits above. But, each commit is not feature-complete, i.e., a single commit doesn't introduce a full feature. So, if the first PR was merged, then Buildbot would have been effectively broken util the third PR was merged. Therefore, we decided to make a single PR but the commits still show the division of code very well.\r\n\r\nInstructions for use:\r\n* Install Buildbot with for development with `pip -e`, fetch and apply this PR to your code.\r\n* Install InfluxDB (Google), go to web interface (typically localhost:8083) and create user `root` with pass `root`. Also create a DB called test.\r\n* In the virtualenv for Buildbot, install influxdb (https://github.com/influxdb/influxdb-python)\r\n* Open master.cfg and add:\r\n```\r\ncaptures = [stats.CaptureProperty('runtests', 'tree-size-KiB')]\r\nc['statsServices'] = [stats.InfluxStorageService('localhost', 8086, 'root', 'root', 'test', captures)]\r\n```\r\nReplace `runtests` with your own builder name and `tree-size-KiB` with a property generated in the builds of this builder.\r\n\r\nThen, run Buildbot as usual (maybe force a build or two). To see the data, go to influx's web interface, log in with username and password. Select the database test. This will bring you to a input prompt for a sql like language. Run `show series;` to get the name of the series. It will be of the form `<builder_name>-<property_name>`. Then, run a select *:\r\n`select * from \"series_name\";`\r\n(Keep the double quotes). This will show you a table of values that Influx registered.\r\n\r\nNote: All tests are pasing. pep8/pyflakes passes, pylint gives some warnings but no errors. Rebased on top of master.","created_at":"2015-06-15T17:10:52Z","updated_at":"2015-07-09T19:19:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9b0cdcb8b265cdfe51f83bc925e7c3ff5b007f4e","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/buildbot/buildbot/pulls/1725/commits","review_comments_url":"https://api.github.com/repos/buildbot/buildbot/pulls/1725/comments","review_comment_url":"https://api.github.com/repos/buildbot/buildbot/pulls/comments{/number}","comments_url":"https://api.github.com/repos/buildbot/buildbot/issues/1725/comments","statuses_url":"https://api.github.com/repos/buildbot/buildbot/statuses/8ba9ac5f74dbda6ad2fac00db1d47f358f69c69d","head":{"label":"prasoon2211:stats-service","ref":"stats-service","sha":"8ba9ac5f74dbda6ad2fac00db1d47f358f69c69d","user":{"login":"prasoon2211","id":2565248,"avatar_url":"https://avatars.githubusercontent.com/u/2565248?v=3","gravatar_id":"","url":"https://api.github.com/users/prasoon2211","html_url":"https://github.com/prasoon2211","followers_url":"https://api.github.com/users/prasoon2211/followers","following_url":"https://api.github.com/users/prasoon2211/following{/other_user}","gists_url":"https://api.github.com/users/prasoon2211/gists{/gist_id}","starred_url":"https://api.github.com/users/prasoon2211/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prasoon2211/subscriptions","organizations_url":"https://api.github.com/users/prasoon2211/orgs","repos_url":"https://api.github.com/users/prasoon2211/repos","events_url":"https://api.github.com/users/prasoon2211/events{/privacy}","received_events_url":"https://api.github.com/users/prasoon2211/received_events","type":"User","site_admin":false},"repo":{"id":31753664,"name":"buildbot","full_name":"prasoon2211/buildbot","owner":{"login":"prasoon2211","id":2565248,"avatar_url":"https://avatars.githubusercontent.com/u/2565248?v=3","gravatar_id":"","url":"https://api.github.com/users/prasoon2211","html_url":"https://github.com/prasoon2211","followers_url":"https://api.github.com/users/prasoon2211/followers","following_url":"https://api.github.com/users/prasoon2211/following{/other_user}","gists_url":"https://api.github.com/users/prasoon2211/gists{/gist_id}","starred_url":"https://api.github.com/users/prasoon2211/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prasoon2211/subscriptions","organizations_url":"https://api.github.com/users/prasoon2211/orgs","repos_url":"https://api.github.com/users/prasoon2211/repos","events_url":"https://api.github.com/users/prasoon2211/events{/privacy}","received_events_url":"https://api.github.com/users/prasoon2211/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/prasoon2211/buildbot","description":"Python-based continuous integration testing framework; send pull requests for your patches!","fork":true,"url":"https://api.github.com/repos/prasoon2211/buildbot","forks_url":"https://api.github.com/repos/prasoon2211/buildbot/forks","keys_url":"https://api.github.com/repos/prasoon2211/buildbot/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prasoon2211/buildbot/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prasoon2211/buildbot/teams","hooks_url":"https://api.github.com/repos/prasoon2211/buildbot/hooks","issue_events_url":"https://api.github.com/repos/prasoon2211/buildbot/issues/events{/number}","events_url":"https://api.github.com/repos/prasoon2211/buildbot/events","assignees_url":"https://api.github.com/repos/prasoon2211/buildbot/assignees{/user}","branches_url":"https://api.github.com/repos/prasoon2211/buildbot/branches{/branch}","tags_url":"https://api.github.com/repos/prasoon2211/buildbot/tags","blobs_url":"https://api.github.com/repos/prasoon2211/buildbot/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prasoon2211/buildbot/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prasoon2211/buildbot/git/refs{/sha}","trees_url":"https://api.github.com/repos/prasoon2211/buildbot/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prasoon2211/buildbot/statuses/{sha}","languages_url":"https://api.github.com/repos/prasoon2211/buildbot/languages","stargazers_url":"https://api.github.com/repos/prasoon2211/buildbot/stargazers","contributors_url":"https://api.github.com/repos/prasoon2211/buildbot/contributors","subscribers_url":"https://api.github.com/repos/prasoon2211/buildbot/subscribers","subscription_url":"https://api.github.com/repos/prasoon2211/buildbot/subscription","commits_url":"https://api.github.com/repos/prasoon2211/buildbot/commits{/sha}","git_commits_url":"https://api.github.com/repos/prasoon2211/buildbot/git/commits{/sha}","comments_url":"https://api.github.com/repos/prasoon2211/buildbot/comments{/number}","issue_comment_url":"https://api.github.com/repos/prasoon2211/buildbot/issues/comments{/number}","contents_url":"https://api.github.com/repos/prasoon2211/buildbot/contents/{+path}","compare_url":"https://api.github.com/repos/prasoon2211/buildbot/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prasoon2211/buildbot/merges","archive_url":"https://api.github.com/repos/prasoon2211/buildbot/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prasoon2211/buildbot/downloads","issues_url":"https://api.github.com/repos/prasoon2211/buildbot/issues{/number}","pulls_url":"https://api.github.com/repos/prasoon2211/buildbot/pulls{/number}","milestones_url":"https://api.github.com/repos/prasoon2211/buildbot/milestones{/number}","notifications_url":"https://api.github.com/repos/prasoon2211/buildbot/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prasoon2211/buildbot/labels{/name}","releases_url":"https://api.github.com/repos/prasoon2211/buildbot/releases{/id}","created_at":"2015-03-06T05:11:29Z","updated_at":"2015-05-26T11:38:56Z","pushed_at":"2015-07-08T20:53:41Z","git_url":"git://github.com/prasoon2211/buildbot.git","ssh_url":"git@github.com:prasoon2211/buildbot.git","clone_url":"https://github.com/prasoon2211/buildbot.git","svn_url":"https://github.com/prasoon2211/buildbot","homepage":"http://buildbot.net","size":36489,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"buildbot:master","ref":"master","sha":"c0ce869a5ddc178cfccfdfba2136c02b1caf8153","user":{"login":"buildbot","id":324515,"avatar_url":"https://avatars.githubusercontent.com/u/324515?v=3","gravatar_id":"","url":"https://api.github.com/users/buildbot","html_url":"https://github.com/buildbot","followers_url":"https://api.github.com/users/buildbot/followers","following_url":"https://api.github.com/users/buildbot/following{/other_user}","gists_url":"https://api.github.com/users/buildbot/gists{/gist_id}","starred_url":"https://api.github.com/users/buildbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buildbot/subscriptions","organizations_url":"https://api.github.com/users/buildbot/orgs","repos_url":"https://api.github.com/users/buildbot/repos","events_url":"https://api.github.com/users/buildbot/events{/privacy}","received_events_url":"https://api.github.com/users/buildbot/received_events","type":"Organization","site_admin":false},"repo":{"id":760165,"name":"buildbot","full_name":"buildbot/buildbot","owner":{"login":"buildbot","id":324515,"avatar_url":"https://avatars.githubusercontent.com/u/324515?v=3","gravatar_id":"","url":"https://api.github.com/users/buildbot","html_url":"https://github.com/buildbot","followers_url":"https://api.github.com/users/buildbot/followers","following_url":"https://api.github.com/users/buildbot/following{/other_user}","gists_url":"https://api.github.com/users/buildbot/gists{/gist_id}","starred_url":"https://api.github.com/users/buildbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buildbot/subscriptions","organizations_url":"https://api.github.com/users/buildbot/orgs","repos_url":"https://api.github.com/users/buildbot/repos","events_url":"https://api.github.com/users/buildbot/events{/privacy}","received_events_url":"https://api.github.com/users/buildbot/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/buildbot/buildbot","description":"Python-based continuous integration testing framework; your pull requests are more than welcome!","fork":false,"url":"https://api.github.com/repos/buildbot/buildbot","forks_url":"https://api.github.com/repos/buildbot/buildbot/forks","keys_url":"https://api.github.com/repos/buildbot/buildbot/keys{/key_id}","collaborators_url":"https://api.github.com/repos/buildbot/buildbot/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/buildbot/buildbot/teams","hooks_url":"https://api.github.com/repos/buildbot/buildbot/hooks","issue_events_url":"https://api.github.com/repos/buildbot/buildbot/issues/events{/number}","events_url":"https://api.github.com/repos/buildbot/buildbot/events","assignees_url":"https://api.github.com/repos/buildbot/buildbot/assignees{/user}","branches_url":"https://api.github.com/repos/buildbot/buildbot/branches{/branch}","tags_url":"https://api.github.com/repos/buildbot/buildbot/tags","blobs_url":"https://api.github.com/repos/buildbot/buildbot/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/buildbot/buildbot/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/buildbot/buildbot/git/refs{/sha}","trees_url":"https://api.github.com/repos/buildbot/buildbot/git/trees{/sha}","statuses_url":"https://api.github.com/repos/buildbot/buildbot/statuses/{sha}","languages_url":"https://api.github.com/repos/buildbot/buildbot/languages","stargazers_url":"https://api.github.com/repos/buildbot/buildbot/stargazers","contributors_url":"https://api.github.com/repos/buildbot/buildbot/contributors","subscribers_url":"https://api.github.com/repos/buildbot/buildbot/subscribers","subscription_url":"https://api.github.com/repos/buildbot/buildbot/subscription","commits_url":"https://api.github.com/repos/buildbot/buildbot/commits{/sha}","git_commits_url":"https://api.github.com/repos/buildbot/buildbot/git/commits{/sha}","comments_url":"https://api.github.com/repos/buildbot/buildbot/comments{/number}","issue_comment_url":"https://api.github.com/repos/buildbot/buildbot/issues/comments{/number}","contents_url":"https://api.github.com/repos/buildbot/buildbot/contents/{+path}","compare_url":"https://api.github.com/repos/buildbot/buildbot/compare/{base}...{head}","merges_url":"https://api.github.com/repos/buildbot/buildbot/merges","archive_url":"https://api.github.com/repos/buildbot/buildbot/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/buildbot/buildbot/downloads","issues_url":"https://api.github.com/repos/buildbot/buildbot/issues{/number}","pulls_url":"https://api.github.com/repos/buildbot/buildbot/pulls{/number}","milestones_url":"https://api.github.com/repos/buildbot/buildbot/milestones{/number}","notifications_url":"https://api.github.com/repos/buildbot/buildbot/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/buildbot/buildbot/labels{/name}","releases_url":"https://api.github.com/repos/buildbot/buildbot/releases{/id}","created_at":"2010-07-06T17:56:53Z","updated_at":"2015-07-09T16:44:18Z","pushed_at":"2015-07-09T16:31:29Z","git_url":"git://github.com/buildbot/buildbot.git","ssh_url":"git@github.com:buildbot/buildbot.git","clone_url":"https://github.com/buildbot/buildbot.git","svn_url":"https://github.com/buildbot/buildbot","homepage":"http://buildbot.net","size":51624,"stargazers_count":1625,"watchers_count":1625,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":727,"mirror_url":null,"open_issues_count":34,"forks":727,"open_issues":34,"watchers":1625,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/buildbot/buildbot/pulls/1725"},"html":{"href":"https://github.com/buildbot/buildbot/pull/1725"},"issue":{"href":"https://api.github.com/repos/buildbot/buildbot/issues/1725"},"comments":{"href":"https://api.github.com/repos/buildbot/buildbot/issues/1725/comments"},"review_comments":{"href":"https://api.github.com/repos/buildbot/buildbot/pulls/1725/comments"},"review_comment":{"href":"https://api.github.com/repos/buildbot/buildbot/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/buildbot/buildbot/pulls/1725/commits"},"statuses":{"href":"https://api.github.com/repos/buildbot/buildbot/statuses/8ba9ac5f74dbda6ad2fac00db1d47f358f69c69d"}}}}