{"action":"created","comment":{"url":"https://api.github.com/repos/disqus/pgshovel/pulls/comments/38688810","id":38688810,"diff_hunk":"@@ -81,36 +129,147 @@ Creating a Cluster\n Running a Relay\n ---------------\n \n+Now, with the replication set all set up, you can run a relay process.\n+\n ::\n \n     pgshovel-relay configuration.yml example\n \n+The configuration file defines the relay object that should be run, as well its configuration.\n+\n+\n+Replication Worker\n+------------------\n+\n+A replication worker is used to replicate a stream of batch operations into a new database replication target. It is started with the ``pgshovel-replicate`` command, passing the path to the replication config file as well as the replicaion set name.\n+\n+::\n+\n+    python-replicate example/configurations/replication.yml example\n+\n+The config file looks very similar to the relay config file, though it contains three sections: ``loader``, ``stream``, and ``target``, corresponding to the three components of replication as described in the \"Concepts\" section above.\n+\n+The replication feature is ongoing, so further documentation of the API is unavailable until the API becomes stable.\n+\n+Results\n+-------\n+\n+That's it!  The tables defined in the ``example`` replication set now have their mutations replicated out of PostgreSQL by the relay.  Additionally, the replication worker bootstrapped a new target and is now applying any new mutations to the target data store.\n+\n+Further Administration\n+======================\n+\n+``pgshovel`` also supports updates to its configuration via a variety of tasks.\n+\n Updating a Replication Set\n --------------------------\n \n+If for any reason you need to update a replication set, you may do so using the ``update`` command, replacing the existing configuration with the one contained in the new config file.\n+\n ::\n \n-    pgshovel-admin set upgrade /path/to/configuration.pgshovel\n+    pgshovel-admin set update /path/to/configuration.pgshovel\n \n Dropping a Replication Set\n --------------------------\n \n+Replication sets may be dropped via the ``drop`` command.\n+\n ::\n \n     pgshovel-admin set drop example\n \n-Configuration\n-=============\n+Further Configuration and Development\n+======================================\n \n-.. todo:: Rewrite with config file\n+In order to run a relay or replication worker, you need to specify a configuration file for their respective command line tool. This section documents the format of those configuration files, as well as information on writing your own components for both systems.\n \n-Operations\n-==========\n+Batch Operations\n+----------------\n \n-Bootstrapping Consumers\n------------------------\n+The relay writes, and the replication stream reads, streams of batch operation messages. An ordered sequence of these operations describe a batch of mutations made against the tables contained in a replication set. A batch operation is a message communicating one of 4 possible actions:\n \n-.. todo:: Rewrite this as part of the tutorial using pgbench after the replication worker is done.\n+1. BeginOperation\n+2. MutationOperation\n+3. CommitOperation\n+4. RollbackOperation\n+\n+A batch of mutations start with a ``BeginOperation``. It is then followed up by zero-to-many ``MutationOperation`` messages, signifying a mutation that was made to a table. The ``MutationOperation``s are followed by either a ``CommitOperation`` signifying the mutation batch was successfully extracted from PostgreSQL.","path":"README.rst","position":193,"original_position":193,"commit_id":"87fb53fe45ae225460ad05665b982337f4f07f73","original_commit_id":"87fb53fe45ae225460ad05665b982337f4f07f73","user":{"login":"tkaemming","id":65315,"avatar_url":"https://avatars.githubusercontent.com/u/65315?v=3","gravatar_id":"","url":"https://api.github.com/users/tkaemming","html_url":"https://github.com/tkaemming","followers_url":"https://api.github.com/users/tkaemming/followers","following_url":"https://api.github.com/users/tkaemming/following{/other_user}","gists_url":"https://api.github.com/users/tkaemming/gists{/gist_id}","starred_url":"https://api.github.com/users/tkaemming/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tkaemming/subscriptions","organizations_url":"https://api.github.com/users/tkaemming/orgs","repos_url":"https://api.github.com/users/tkaemming/repos","events_url":"https://api.github.com/users/tkaemming/events{/privacy}","received_events_url":"https://api.github.com/users/tkaemming/received_events","type":"User","site_admin":false},"body":"This renders weird, not sure why.","created_at":"2015-09-03T19:53:12Z","updated_at":"2015-09-03T19:53:12Z","html_url":"https://github.com/disqus/pgshovel/pull/25#discussion_r38688810","pull_request_url":"https://api.github.com/repos/disqus/pgshovel/pulls/25","_links":{"self":{"href":"https://api.github.com/repos/disqus/pgshovel/pulls/comments/38688810"},"html":{"href":"https://github.com/disqus/pgshovel/pull/25#discussion_r38688810"},"pull_request":{"href":"https://api.github.com/repos/disqus/pgshovel/pulls/25"}}},"pull_request":{"url":"https://api.github.com/repos/disqus/pgshovel/pulls/25","id":43677572,"html_url":"https://github.com/disqus/pgshovel/pull/25","diff_url":"https://github.com/disqus/pgshovel/pull/25.diff","patch_url":"https://github.com/disqus/pgshovel/pull/25.patch","issue_url":"https://api.github.com/repos/disqus/pgshovel/issues/25","number":25,"state":"open","locked":false,"title":"Modernize documentation for replication","user":{"login":"Fluxx","id":5892,"avatar_url":"https://avatars.githubusercontent.com/u/5892?v=3","gravatar_id":"","url":"https://api.github.com/users/Fluxx","html_url":"https://github.com/Fluxx","followers_url":"https://api.github.com/users/Fluxx/followers","following_url":"https://api.github.com/users/Fluxx/following{/other_user}","gists_url":"https://api.github.com/users/Fluxx/gists{/gist_id}","starred_url":"https://api.github.com/users/Fluxx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fluxx/subscriptions","organizations_url":"https://api.github.com/users/Fluxx/orgs","repos_url":"https://api.github.com/users/Fluxx/repos","events_url":"https://api.github.com/users/Fluxx/events{/privacy}","received_events_url":"https://api.github.com/users/Fluxx/received_events","type":"User","site_admin":false},"body":"I went through the README and updated its documentation to be a little richer, more exhaustive and also talk about the emerging replication branch.  This is very much a draft, so there are likely spelling and grammar errors. Please point those out but do expect them.","created_at":"2015-08-28T22:16:22Z","updated_at":"2015-09-03T19:53:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":"004599f3589a6300634eb547fb8f4a4d6c68fe85","assignee":null,"milestone":{"url":"https://api.github.com/repos/disqus/pgshovel/milestones/2","html_url":"https://github.com/disqus/pgshovel/milestones/0.9","labels_url":"https://api.github.com/repos/disqus/pgshovel/milestones/2/labels","id":1277982,"number":2,"title":"0.9","description":"Represents a version containing replication tooling and no major known bugs, but with some known issues (mostly administrative and/or operational in nature) intended to be used for non-mission critical use cases where hours (or days) of downtime and potential data corruption (possibly irrecoverable) due to operator error can occur.","creator":{"login":"Fluxx","id":5892,"avatar_url":"https://avatars.githubusercontent.com/u/5892?v=3","gravatar_id":"","url":"https://api.github.com/users/Fluxx","html_url":"https://github.com/Fluxx","followers_url":"https://api.github.com/users/Fluxx/followers","following_url":"https://api.github.com/users/Fluxx/following{/other_user}","gists_url":"https://api.github.com/users/Fluxx/gists{/gist_id}","starred_url":"https://api.github.com/users/Fluxx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fluxx/subscriptions","organizations_url":"https://api.github.com/users/Fluxx/orgs","repos_url":"https://api.github.com/users/Fluxx/repos","events_url":"https://api.github.com/users/Fluxx/events{/privacy}","received_events_url":"https://api.github.com/users/Fluxx/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":2,"state":"open","created_at":"2015-08-28T17:20:08Z","updated_at":"2015-09-02T16:03:06Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/disqus/pgshovel/pulls/25/commits","review_comments_url":"https://api.github.com/repos/disqus/pgshovel/pulls/25/comments","review_comment_url":"https://api.github.com/repos/disqus/pgshovel/pulls/comments{/number}","comments_url":"https://api.github.com/repos/disqus/pgshovel/issues/25/comments","statuses_url":"https://api.github.com/repos/disqus/pgshovel/statuses/87fb53fe45ae225460ad05665b982337f4f07f73","head":{"label":"disqus:replication_documentation","ref":"replication_documentation","sha":"87fb53fe45ae225460ad05665b982337f4f07f73","user":{"login":"disqus","id":317745,"avatar_url":"https://avatars.githubusercontent.com/u/317745?v=3","gravatar_id":"","url":"https://api.github.com/users/disqus","html_url":"https://github.com/disqus","followers_url":"https://api.github.com/users/disqus/followers","following_url":"https://api.github.com/users/disqus/following{/other_user}","gists_url":"https://api.github.com/users/disqus/gists{/gist_id}","starred_url":"https://api.github.com/users/disqus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/disqus/subscriptions","organizations_url":"https://api.github.com/users/disqus/orgs","repos_url":"https://api.github.com/users/disqus/repos","events_url":"https://api.github.com/users/disqus/events{/privacy}","received_events_url":"https://api.github.com/users/disqus/received_events","type":"Organization","site_admin":false},"repo":{"id":40566869,"name":"pgshovel","full_name":"disqus/pgshovel","owner":{"login":"disqus","id":317745,"avatar_url":"https://avatars.githubusercontent.com/u/317745?v=3","gravatar_id":"","url":"https://api.github.com/users/disqus","html_url":"https://github.com/disqus","followers_url":"https://api.github.com/users/disqus/followers","following_url":"https://api.github.com/users/disqus/following{/other_user}","gists_url":"https://api.github.com/users/disqus/gists{/gist_id}","starred_url":"https://api.github.com/users/disqus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/disqus/subscriptions","organizations_url":"https://api.github.com/users/disqus/orgs","repos_url":"https://api.github.com/users/disqus/repos","events_url":"https://api.github.com/users/disqus/events{/privacy}","received_events_url":"https://api.github.com/users/disqus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/disqus/pgshovel","description":"A change data capture system for PostgreSQL","fork":false,"url":"https://api.github.com/repos/disqus/pgshovel","forks_url":"https://api.github.com/repos/disqus/pgshovel/forks","keys_url":"https://api.github.com/repos/disqus/pgshovel/keys{/key_id}","collaborators_url":"https://api.github.com/repos/disqus/pgshovel/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/disqus/pgshovel/teams","hooks_url":"https://api.github.com/repos/disqus/pgshovel/hooks","issue_events_url":"https://api.github.com/repos/disqus/pgshovel/issues/events{/number}","events_url":"https://api.github.com/repos/disqus/pgshovel/events","assignees_url":"https://api.github.com/repos/disqus/pgshovel/assignees{/user}","branches_url":"https://api.github.com/repos/disqus/pgshovel/branches{/branch}","tags_url":"https://api.github.com/repos/disqus/pgshovel/tags","blobs_url":"https://api.github.com/repos/disqus/pgshovel/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/disqus/pgshovel/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/disqus/pgshovel/git/refs{/sha}","trees_url":"https://api.github.com/repos/disqus/pgshovel/git/trees{/sha}","statuses_url":"https://api.github.com/repos/disqus/pgshovel/statuses/{sha}","languages_url":"https://api.github.com/repos/disqus/pgshovel/languages","stargazers_url":"https://api.github.com/repos/disqus/pgshovel/stargazers","contributors_url":"https://api.github.com/repos/disqus/pgshovel/contributors","subscribers_url":"https://api.github.com/repos/disqus/pgshovel/subscribers","subscription_url":"https://api.github.com/repos/disqus/pgshovel/subscription","commits_url":"https://api.github.com/repos/disqus/pgshovel/commits{/sha}","git_commits_url":"https://api.github.com/repos/disqus/pgshovel/git/commits{/sha}","comments_url":"https://api.github.com/repos/disqus/pgshovel/comments{/number}","issue_comment_url":"https://api.github.com/repos/disqus/pgshovel/issues/comments{/number}","contents_url":"https://api.github.com/repos/disqus/pgshovel/contents/{+path}","compare_url":"https://api.github.com/repos/disqus/pgshovel/compare/{base}...{head}","merges_url":"https://api.github.com/repos/disqus/pgshovel/merges","archive_url":"https://api.github.com/repos/disqus/pgshovel/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/disqus/pgshovel/downloads","issues_url":"https://api.github.com/repos/disqus/pgshovel/issues{/number}","pulls_url":"https://api.github.com/repos/disqus/pgshovel/pulls{/number}","milestones_url":"https://api.github.com/repos/disqus/pgshovel/milestones{/number}","notifications_url":"https://api.github.com/repos/disqus/pgshovel/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/disqus/pgshovel/labels{/name}","releases_url":"https://api.github.com/repos/disqus/pgshovel/releases{/id}","created_at":"2015-08-11T21:49:27Z","updated_at":"2015-09-02T15:26:36Z","pushed_at":"2015-09-03T19:05:23Z","git_url":"git://github.com/disqus/pgshovel.git","ssh_url":"git@github.com:disqus/pgshovel.git","clone_url":"https://github.com/disqus/pgshovel.git","svn_url":"https://github.com/disqus/pgshovel","homepage":"","size":676,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":24,"forks":1,"open_issues":24,"watchers":2,"default_branch":"master"}},"base":{"label":"disqus:replication","ref":"replication","sha":"1f18a7b75494bdeabd8f78b872f50560a91f9951","user":{"login":"disqus","id":317745,"avatar_url":"https://avatars.githubusercontent.com/u/317745?v=3","gravatar_id":"","url":"https://api.github.com/users/disqus","html_url":"https://github.com/disqus","followers_url":"https://api.github.com/users/disqus/followers","following_url":"https://api.github.com/users/disqus/following{/other_user}","gists_url":"https://api.github.com/users/disqus/gists{/gist_id}","starred_url":"https://api.github.com/users/disqus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/disqus/subscriptions","organizations_url":"https://api.github.com/users/disqus/orgs","repos_url":"https://api.github.com/users/disqus/repos","events_url":"https://api.github.com/users/disqus/events{/privacy}","received_events_url":"https://api.github.com/users/disqus/received_events","type":"Organization","site_admin":false},"repo":{"id":40566869,"name":"pgshovel","full_name":"disqus/pgshovel","owner":{"login":"disqus","id":317745,"avatar_url":"https://avatars.githubusercontent.com/u/317745?v=3","gravatar_id":"","url":"https://api.github.com/users/disqus","html_url":"https://github.com/disqus","followers_url":"https://api.github.com/users/disqus/followers","following_url":"https://api.github.com/users/disqus/following{/other_user}","gists_url":"https://api.github.com/users/disqus/gists{/gist_id}","starred_url":"https://api.github.com/users/disqus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/disqus/subscriptions","organizations_url":"https://api.github.com/users/disqus/orgs","repos_url":"https://api.github.com/users/disqus/repos","events_url":"https://api.github.com/users/disqus/events{/privacy}","received_events_url":"https://api.github.com/users/disqus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/disqus/pgshovel","description":"A change data capture system for PostgreSQL","fork":false,"url":"https://api.github.com/repos/disqus/pgshovel","forks_url":"https://api.github.com/repos/disqus/pgshovel/forks","keys_url":"https://api.github.com/repos/disqus/pgshovel/keys{/key_id}","collaborators_url":"https://api.github.com/repos/disqus/pgshovel/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/disqus/pgshovel/teams","hooks_url":"https://api.github.com/repos/disqus/pgshovel/hooks","issue_events_url":"https://api.github.com/repos/disqus/pgshovel/issues/events{/number}","events_url":"https://api.github.com/repos/disqus/pgshovel/events","assignees_url":"https://api.github.com/repos/disqus/pgshovel/assignees{/user}","branches_url":"https://api.github.com/repos/disqus/pgshovel/branches{/branch}","tags_url":"https://api.github.com/repos/disqus/pgshovel/tags","blobs_url":"https://api.github.com/repos/disqus/pgshovel/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/disqus/pgshovel/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/disqus/pgshovel/git/refs{/sha}","trees_url":"https://api.github.com/repos/disqus/pgshovel/git/trees{/sha}","statuses_url":"https://api.github.com/repos/disqus/pgshovel/statuses/{sha}","languages_url":"https://api.github.com/repos/disqus/pgshovel/languages","stargazers_url":"https://api.github.com/repos/disqus/pgshovel/stargazers","contributors_url":"https://api.github.com/repos/disqus/pgshovel/contributors","subscribers_url":"https://api.github.com/repos/disqus/pgshovel/subscribers","subscription_url":"https://api.github.com/repos/disqus/pgshovel/subscription","commits_url":"https://api.github.com/repos/disqus/pgshovel/commits{/sha}","git_commits_url":"https://api.github.com/repos/disqus/pgshovel/git/commits{/sha}","comments_url":"https://api.github.com/repos/disqus/pgshovel/comments{/number}","issue_comment_url":"https://api.github.com/repos/disqus/pgshovel/issues/comments{/number}","contents_url":"https://api.github.com/repos/disqus/pgshovel/contents/{+path}","compare_url":"https://api.github.com/repos/disqus/pgshovel/compare/{base}...{head}","merges_url":"https://api.github.com/repos/disqus/pgshovel/merges","archive_url":"https://api.github.com/repos/disqus/pgshovel/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/disqus/pgshovel/downloads","issues_url":"https://api.github.com/repos/disqus/pgshovel/issues{/number}","pulls_url":"https://api.github.com/repos/disqus/pgshovel/pulls{/number}","milestones_url":"https://api.github.com/repos/disqus/pgshovel/milestones{/number}","notifications_url":"https://api.github.com/repos/disqus/pgshovel/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/disqus/pgshovel/labels{/name}","releases_url":"https://api.github.com/repos/disqus/pgshovel/releases{/id}","created_at":"2015-08-11T21:49:27Z","updated_at":"2015-09-02T15:26:36Z","pushed_at":"2015-09-03T19:05:23Z","git_url":"git://github.com/disqus/pgshovel.git","ssh_url":"git@github.com:disqus/pgshovel.git","clone_url":"https://github.com/disqus/pgshovel.git","svn_url":"https://github.com/disqus/pgshovel","homepage":"","size":676,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":24,"forks":1,"open_issues":24,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/disqus/pgshovel/pulls/25"},"html":{"href":"https://github.com/disqus/pgshovel/pull/25"},"issue":{"href":"https://api.github.com/repos/disqus/pgshovel/issues/25"},"comments":{"href":"https://api.github.com/repos/disqus/pgshovel/issues/25/comments"},"review_comments":{"href":"https://api.github.com/repos/disqus/pgshovel/pulls/25/comments"},"review_comment":{"href":"https://api.github.com/repos/disqus/pgshovel/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/disqus/pgshovel/pulls/25/commits"},"statuses":{"href":"https://api.github.com/repos/disqus/pgshovel/statuses/87fb53fe45ae225460ad05665b982337f4f07f73"}}}}