{"action":"created","comment":{"url":"https://api.github.com/repos/facebook/osquery/pulls/comments/38940385","id":38940385,"diff_hunk":"@@ -83,6 +83,61 @@ The posted logger data is exactly the same as logged to disk by the **filesystem\n {}\n ```\n \n+**Distributed** read request POST body:\n+\n+```json\n+{\n+  \"node_key\": \"...\"\n+}\n+```\n+\n+**Distributed** read responde POST body:\n+\n+```json\n+{\n+    \"queries\": {\n+        \"id1\": \"select * from osquery_info;\",\n+        \"id2\": \"select * from osquery_schedule;\"\n+    }\n+}\n+```\n+\n+**Distributed** write request POST body:\n+\n+```json\n+{\n+    \"node_key\": \"...\",\n+    \"queries\": {\n+        \"id1\": [\n+            {\n+                \"colum1\": \"value1\",","path":"docs/wiki/deployment/remote.md","position":null,"original_position":31,"commit_id":"ee90dcfbe9d2065cebbf11868f7becf79af1dd99","original_commit_id":"3c15ee9ee017063b3a890e64a3a7a82ae39c4585","user":{"login":"marpaia","id":927168,"avatar_url":"https://avatars.githubusercontent.com/u/927168?v=3","gravatar_id":"","url":"https://api.github.com/users/marpaia","html_url":"https://github.com/marpaia","followers_url":"https://api.github.com/users/marpaia/followers","following_url":"https://api.github.com/users/marpaia/following{/other_user}","gists_url":"https://api.github.com/users/marpaia/gists{/gist_id}","starred_url":"https://api.github.com/users/marpaia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marpaia/subscriptions","organizations_url":"https://api.github.com/users/marpaia/orgs","repos_url":"https://api.github.com/users/marpaia/repos","events_url":"https://api.github.com/users/marpaia/events{/privacy}","received_events_url":"https://api.github.com/users/marpaia/received_events","type":"User","site_admin":false},"body":"danka :)","created_at":"2015-09-08T15:33:01Z","updated_at":"2015-09-08T15:33:01Z","html_url":"https://github.com/facebook/osquery/pull/1500#discussion_r38940385","pull_request_url":"https://api.github.com/repos/facebook/osquery/pulls/1500","_links":{"self":{"href":"https://api.github.com/repos/facebook/osquery/pulls/comments/38940385"},"html":{"href":"https://github.com/facebook/osquery/pull/1500#discussion_r38940385"},"pull_request":{"href":"https://api.github.com/repos/facebook/osquery/pulls/1500"}}},"pull_request":{"url":"https://api.github.com/repos/facebook/osquery/pulls/1500","id":44379483,"html_url":"https://github.com/facebook/osquery/pull/1500","diff_url":"https://github.com/facebook/osquery/pull/1500.diff","patch_url":"https://github.com/facebook/osquery/pull/1500.patch","issue_url":"https://api.github.com/repos/facebook/osquery/issues/1500","number":1500,"state":"open","locked":false,"title":"Client-side implementation of distributed queries","user":{"login":"marpaia","id":927168,"avatar_url":"https://avatars.githubusercontent.com/u/927168?v=3","gravatar_id":"","url":"https://api.github.com/users/marpaia","html_url":"https://github.com/marpaia","followers_url":"https://api.github.com/users/marpaia/followers","following_url":"https://api.github.com/users/marpaia/following{/other_user}","gists_url":"https://api.github.com/users/marpaia/gists{/gist_id}","starred_url":"https://api.github.com/users/marpaia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marpaia/subscriptions","organizations_url":"https://api.github.com/users/marpaia/orgs","repos_url":"https://api.github.com/users/marpaia/repos","events_url":"https://api.github.com/users/marpaia/events{/privacy}","received_events_url":"https://api.github.com/users/marpaia/received_events","type":"User","site_admin":false},"body":"This pull request contains the client-side implementation of remote distributed queries for osqueryd. For example, using an appropriate backend one could easily distribute queries to their entire fleet (or an arbitrary subset thereof) and digest the results out of band of the normal logging pipeline.\r\n\r\nAlso included is an example `tls` plugin. It follows the same paradigms as the existing config and logger `tls` plugins, so adding support to existing osquery backends, like https://github.com/blackfist/windmill, should be straightforward.\r\n\r\n### The following flags are added with this diff\r\n\r\n`--distributed_plugin=tls`\r\n\r\nDistributed plugin name. The default distributed plugin is not set. You must set `distributed_enabled` and `distributed_plugin` to use the distributed feature.\r\n\r\n`--distributed_enabled=false`\r\n\r\nMain killswitch for distributed queries functionality. By default, this is turned off. If you want to turn on distributed queries, you must enable this flag as well as specify the plugin that you'd like to use via the `distributed_plugin` flag.\r\n\r\n`--distributed_poll_interval=60`\r\n\r\nIn seconds, the amount of time that osqueryd will wait between periodically checking in with a distributed query server to see if there are any queries to execute.\r\n\r\n`--distributed_tls_read_endpoint=\"\"`\r\n\r\nThe TLS endpoint path, e.g: `/api/v1/distributed/read` when using the `tls` distributed plugin. See the other `tls` related CLI flags.\r\n\r\n`--distributed_tls_write_endpoint=\"\"`\r\n\r\nThe TLS endpoint path, e.g: `/api/v1/distributed/write` when using the `tls` distributed plugin. See the other `tls` related CLI flags.\r\n\r\n`--distributed_tls_max_attempts=3`\r\n\r\nThe total number of attempts that will be made to the remote distributed query server if a request fails when using the `tls` distributed plugin.\r\n\r\n### plugins!\r\n\r\nThe `DistributedPlugin` interface is as follows:\r\n\r\n```cpp\r\nnamespace osquery {\r\n\r\nclass DistributedPlugin : public Plugin {\r\n public:\r\n  virtual Status getQueries(std::string& json) = 0;\r\n  virtual Status writeResults(const std::string& json) = 0;\r\n};\r\n}\r\n```\r\n\r\n`include/osquery/distributed.h` has extensive details on the JSON format which have been redacted here for brevity.\r\n\r\nTo create a custom plugin, subclass `DistributedPlugin` and then, as usual:\r\n\r\n```cpp\r\nREGISTER(CustomDistributedPlugin, \"distributed\", \"custom\");\r\n```\r\n\r\n### TLS request/response format\r\n\r\nAs noted above, a `tls` plugin is included for the `distributed` plugin interface. The request/response format which is included in the documentation, as well as the doxygen comments in the code is repeated here:\r\n\r\n**Distributed** read request POST body:\r\n\r\n```json\r\n{\r\n  \"node_key\": \"...\"\r\n}\r\n```\r\n\r\n**Distributed** read responde POST body:\r\n\r\n```json\r\n{\r\n    \"queries\": {\r\n        \"id1\": \"select * from osquery_info;\",\r\n        \"id2\": \"select * from osquery_schedule;\"\r\n    }\r\n}\r\n```\r\n\r\n**Distributed** write request POST body:\r\n\r\n```json\r\n{\r\n    \"node_key\": \"...\",\r\n    \"queries\": {\r\n        \"id1\": [\r\n            {\r\n                \"colum1\": \"value1\",\r\n                \"colum2\": \"value2\"\r\n            },\r\n            {\r\n                \"colum1\": \"value1\",\r\n                \"colum2\": \"value2\"\r\n            }\r\n        ],\r\n        \"id2\": [\r\n            {\r\n                \"colum1\": \"value1\",\r\n                \"colum2\": \"value2\"\r\n            },\r\n            {\r\n                \"colum1\": \"value1\",\r\n                \"colum2\": \"value2\"\r\n            }\r\n        ]\r\n    }\r\n}\r\n```\r\n\r\n**Distributed** write request POST body:\r\n\r\n```json\r\n{}\r\n```\r\n\r\n### additional things in here\r\n\r\nSince I was in the `tls` plugins, I refactored a bit so that I could share more code from the config tls plugin. As a result, I added one additional config flag for the `tls` config plugin:\r\n\r\n`--config_tls_max_attempts=3`\r\n\r\nThe total number of attempts that will be made to the remote config server if a request fails.\r\n\r\n---\r\n\r\nI moved `include/osquery/config/packs.h` to `include/osquery/packs.h`. It was the only header in `include/osquery` in a subdirectory and it really didn't need to be. There were only three implementation files which included packs.h, so they were easy to upgrade and sneak in here.\r\n\r\n---\r\n\r\nI added default values for the thrift macros that we use to maintain internal and external include paths. This doesn't functionally change anything other than make the include paths more sane when not parsed with our build system.","created_at":"2015-09-07T21:14:41Z","updated_at":"2015-09-08T15:33:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"90709dd3a3bd5af913345b79de6982fd975a64e0","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/facebook/osquery/pulls/1500/commits","review_comments_url":"https://api.github.com/repos/facebook/osquery/pulls/1500/comments","review_comment_url":"https://api.github.com/repos/facebook/osquery/pulls/comments{/number}","comments_url":"https://api.github.com/repos/facebook/osquery/issues/1500/comments","statuses_url":"https://api.github.com/repos/facebook/osquery/statuses/ee90dcfbe9d2065cebbf11868f7becf79af1dd99","head":{"label":"marpaia:remote","ref":"remote","sha":"ee90dcfbe9d2065cebbf11868f7becf79af1dd99","user":{"login":"marpaia","id":927168,"avatar_url":"https://avatars.githubusercontent.com/u/927168?v=3","gravatar_id":"","url":"https://api.github.com/users/marpaia","html_url":"https://github.com/marpaia","followers_url":"https://api.github.com/users/marpaia/followers","following_url":"https://api.github.com/users/marpaia/following{/other_user}","gists_url":"https://api.github.com/users/marpaia/gists{/gist_id}","starred_url":"https://api.github.com/users/marpaia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marpaia/subscriptions","organizations_url":"https://api.github.com/users/marpaia/orgs","repos_url":"https://api.github.com/users/marpaia/repos","events_url":"https://api.github.com/users/marpaia/events{/privacy}","received_events_url":"https://api.github.com/users/marpaia/received_events","type":"User","site_admin":false},"repo":{"id":29230929,"name":"osquery","full_name":"marpaia/osquery","owner":{"login":"marpaia","id":927168,"avatar_url":"https://avatars.githubusercontent.com/u/927168?v=3","gravatar_id":"","url":"https://api.github.com/users/marpaia","html_url":"https://github.com/marpaia","followers_url":"https://api.github.com/users/marpaia/followers","following_url":"https://api.github.com/users/marpaia/following{/other_user}","gists_url":"https://api.github.com/users/marpaia/gists{/gist_id}","starred_url":"https://api.github.com/users/marpaia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marpaia/subscriptions","organizations_url":"https://api.github.com/users/marpaia/orgs","repos_url":"https://api.github.com/users/marpaia/repos","events_url":"https://api.github.com/users/marpaia/events{/privacy}","received_events_url":"https://api.github.com/users/marpaia/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/marpaia/osquery","description":"SQL powered operating system instrumentation, monitoring, and analytics.","fork":true,"url":"https://api.github.com/repos/marpaia/osquery","forks_url":"https://api.github.com/repos/marpaia/osquery/forks","keys_url":"https://api.github.com/repos/marpaia/osquery/keys{/key_id}","collaborators_url":"https://api.github.com/repos/marpaia/osquery/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/marpaia/osquery/teams","hooks_url":"https://api.github.com/repos/marpaia/osquery/hooks","issue_events_url":"https://api.github.com/repos/marpaia/osquery/issues/events{/number}","events_url":"https://api.github.com/repos/marpaia/osquery/events","assignees_url":"https://api.github.com/repos/marpaia/osquery/assignees{/user}","branches_url":"https://api.github.com/repos/marpaia/osquery/branches{/branch}","tags_url":"https://api.github.com/repos/marpaia/osquery/tags","blobs_url":"https://api.github.com/repos/marpaia/osquery/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/marpaia/osquery/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/marpaia/osquery/git/refs{/sha}","trees_url":"https://api.github.com/repos/marpaia/osquery/git/trees{/sha}","statuses_url":"https://api.github.com/repos/marpaia/osquery/statuses/{sha}","languages_url":"https://api.github.com/repos/marpaia/osquery/languages","stargazers_url":"https://api.github.com/repos/marpaia/osquery/stargazers","contributors_url":"https://api.github.com/repos/marpaia/osquery/contributors","subscribers_url":"https://api.github.com/repos/marpaia/osquery/subscribers","subscription_url":"https://api.github.com/repos/marpaia/osquery/subscription","commits_url":"https://api.github.com/repos/marpaia/osquery/commits{/sha}","git_commits_url":"https://api.github.com/repos/marpaia/osquery/git/commits{/sha}","comments_url":"https://api.github.com/repos/marpaia/osquery/comments{/number}","issue_comment_url":"https://api.github.com/repos/marpaia/osquery/issues/comments{/number}","contents_url":"https://api.github.com/repos/marpaia/osquery/contents/{+path}","compare_url":"https://api.github.com/repos/marpaia/osquery/compare/{base}...{head}","merges_url":"https://api.github.com/repos/marpaia/osquery/merges","archive_url":"https://api.github.com/repos/marpaia/osquery/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/marpaia/osquery/downloads","issues_url":"https://api.github.com/repos/marpaia/osquery/issues{/number}","pulls_url":"https://api.github.com/repos/marpaia/osquery/pulls{/number}","milestones_url":"https://api.github.com/repos/marpaia/osquery/milestones{/number}","notifications_url":"https://api.github.com/repos/marpaia/osquery/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/marpaia/osquery/labels{/name}","releases_url":"https://api.github.com/repos/marpaia/osquery/releases{/id}","created_at":"2015-01-14T06:36:21Z","updated_at":"2015-06-26T00:54:20Z","pushed_at":"2015-09-08T15:30:58Z","git_url":"git://github.com/marpaia/osquery.git","ssh_url":"git@github.com:marpaia/osquery.git","clone_url":"https://github.com/marpaia/osquery.git","svn_url":"https://github.com/marpaia/osquery","homepage":"https://osquery.io","size":6367,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"facebook:master","ref":"master","sha":"2a20804f59ee3d41357f9a032f6e23da75a3245b","user":{"login":"facebook","id":69631,"avatar_url":"https://avatars.githubusercontent.com/u/69631?v=3","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"repo":{"id":22394357,"name":"osquery","full_name":"facebook/osquery","owner":{"login":"facebook","id":69631,"avatar_url":"https://avatars.githubusercontent.com/u/69631?v=3","gravatar_id":"","url":"https://api.github.com/users/facebook","html_url":"https://github.com/facebook","followers_url":"https://api.github.com/users/facebook/followers","following_url":"https://api.github.com/users/facebook/following{/other_user}","gists_url":"https://api.github.com/users/facebook/gists{/gist_id}","starred_url":"https://api.github.com/users/facebook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/facebook/subscriptions","organizations_url":"https://api.github.com/users/facebook/orgs","repos_url":"https://api.github.com/users/facebook/repos","events_url":"https://api.github.com/users/facebook/events{/privacy}","received_events_url":"https://api.github.com/users/facebook/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/facebook/osquery","description":"SQL powered operating system instrumentation, monitoring, and analytics.","fork":false,"url":"https://api.github.com/repos/facebook/osquery","forks_url":"https://api.github.com/repos/facebook/osquery/forks","keys_url":"https://api.github.com/repos/facebook/osquery/keys{/key_id}","collaborators_url":"https://api.github.com/repos/facebook/osquery/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/facebook/osquery/teams","hooks_url":"https://api.github.com/repos/facebook/osquery/hooks","issue_events_url":"https://api.github.com/repos/facebook/osquery/issues/events{/number}","events_url":"https://api.github.com/repos/facebook/osquery/events","assignees_url":"https://api.github.com/repos/facebook/osquery/assignees{/user}","branches_url":"https://api.github.com/repos/facebook/osquery/branches{/branch}","tags_url":"https://api.github.com/repos/facebook/osquery/tags","blobs_url":"https://api.github.com/repos/facebook/osquery/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/facebook/osquery/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/facebook/osquery/git/refs{/sha}","trees_url":"https://api.github.com/repos/facebook/osquery/git/trees{/sha}","statuses_url":"https://api.github.com/repos/facebook/osquery/statuses/{sha}","languages_url":"https://api.github.com/repos/facebook/osquery/languages","stargazers_url":"https://api.github.com/repos/facebook/osquery/stargazers","contributors_url":"https://api.github.com/repos/facebook/osquery/contributors","subscribers_url":"https://api.github.com/repos/facebook/osquery/subscribers","subscription_url":"https://api.github.com/repos/facebook/osquery/subscription","commits_url":"https://api.github.com/repos/facebook/osquery/commits{/sha}","git_commits_url":"https://api.github.com/repos/facebook/osquery/git/commits{/sha}","comments_url":"https://api.github.com/repos/facebook/osquery/comments{/number}","issue_comment_url":"https://api.github.com/repos/facebook/osquery/issues/comments{/number}","contents_url":"https://api.github.com/repos/facebook/osquery/contents/{+path}","compare_url":"https://api.github.com/repos/facebook/osquery/compare/{base}...{head}","merges_url":"https://api.github.com/repos/facebook/osquery/merges","archive_url":"https://api.github.com/repos/facebook/osquery/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/facebook/osquery/downloads","issues_url":"https://api.github.com/repos/facebook/osquery/issues{/number}","pulls_url":"https://api.github.com/repos/facebook/osquery/pulls{/number}","milestones_url":"https://api.github.com/repos/facebook/osquery/milestones{/number}","notifications_url":"https://api.github.com/repos/facebook/osquery/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/facebook/osquery/labels{/name}","releases_url":"https://api.github.com/repos/facebook/osquery/releases{/id}","created_at":"2014-07-29T20:27:33Z","updated_at":"2015-09-08T13:57:41Z","pushed_at":"2015-09-08T15:30:59Z","git_url":"git://github.com/facebook/osquery.git","ssh_url":"git@github.com:facebook/osquery.git","clone_url":"https://github.com/facebook/osquery.git","svn_url":"https://github.com/facebook/osquery","homepage":"https://osquery.io","size":15795,"stargazers_count":5008,"watchers_count":5008,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":444,"mirror_url":null,"open_issues_count":16,"forks":444,"open_issues":16,"watchers":5008,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/facebook/osquery/pulls/1500"},"html":{"href":"https://github.com/facebook/osquery/pull/1500"},"issue":{"href":"https://api.github.com/repos/facebook/osquery/issues/1500"},"comments":{"href":"https://api.github.com/repos/facebook/osquery/issues/1500/comments"},"review_comments":{"href":"https://api.github.com/repos/facebook/osquery/pulls/1500/comments"},"review_comment":{"href":"https://api.github.com/repos/facebook/osquery/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/facebook/osquery/pulls/1500/commits"},"statuses":{"href":"https://api.github.com/repos/facebook/osquery/statuses/ee90dcfbe9d2065cebbf11868f7becf79af1dd99"}}}}