{"action":"created","comment":{"url":"https://api.github.com/repos/caskdata/cdap/pulls/comments/23733601","id":23733601,"diff_hunk":"@@ -0,0 +1,140 @@\n+.. meta::\n+    :author: Cask Data, Inc.\n+    :copyright: Copyright © 2015 Cask Data, Inc.\n+\n+================\n+Adapters\n+================\n+\n+Adapters read data from one or more data sources, perform a transformation, then\n+write to one or more data sinks. They may be used for ETL purposes, or any situation\n+where you want to pipe data from one data source to another.\n+\n+CDAP currently provides a stream-conversion adapter that regularly reads data from\n+a Stream, then writes the data to a TimePartitionedFileSet in avro format. The output\n+can then be explored through Hive or Impala, assuming a compatible schema is used. \n+\n+Each Adapter type is implemented as a CDAP Application, with each Adapter instance as\n+a schedule for running a program in the application. For example, the stream-conversion\n+Adapter type is an application named StreamConversion. Each Adapter instance is a schedule\n+that periodically runs the Workflow defined in the StreamConversion application.\n+Custom adapters can be added to CDAP, though the feature is experimental and will\n+likely change and improve in future releases.\n+\n+Stream Conversion Adapter\n+=========================\n+The stream-conversion Adapter type regularly reads data from a source Stream and writes\n+it to a TimePartitionedFileSet as avro files. You may want to use this Adapter if you\n+want to run partitioned Hive or Impala queries on data in a Stream, or if you have set a\n+TTL on a Stream for real-time processing, but want to keep all Stream data around in\n+another format for archiving purposes. Each Stream event in translated as a line in\n+the output fileset. Stream format and schema must be given as adapter properties.\n+The output schema will add a required ``ts`` field of type long for the timestamp of the\n+event, as well as nullable string fields for any headers given in the adapter properties. \n+\n+Adapters of the stream-conversion type require the following adapter properties:\n+\n+  * frequency: how often the Adapter should run, as a number followed by a number. For\n+    example, 5m means the Adapter will run every five minutes, and 2h means the Adapter\n+    will run every two hours. Note that this must be a number that divides evenly into\n+    the time unit given. For example, 30m is valid, but 13m is not because an hour does\n+    not evenly divide into 13 minute chunks.\n+  * source.name: the name of the Stream to read from.\n+  * source.format.name: the name of the format of the Stream body.\n+  * source.format.settings: settings of the format as a JSON Object of string to string.\n+  * source.schema: the schema of the Stream body.\n+  * sink.name: the name of the TimePartitionedFileSet to write to. \n+\n+The following adapter properties are optional:","path":"cdap-docs/developers-manual/source/advanced/adapters.rst","position":48,"original_position":48,"commit_id":"abaef5bee1a836b66215eca60a4649054cdf386c","original_commit_id":"abaef5bee1a836b66215eca60a4649054cdf386c","user":{"login":"awholegunch","id":6394794,"avatar_url":"https://avatars.githubusercontent.com/u/6394794?v=3","gravatar_id":"","url":"https://api.github.com/users/awholegunch","html_url":"https://github.com/awholegunch","followers_url":"https://api.github.com/users/awholegunch/followers","following_url":"https://api.github.com/users/awholegunch/following{/other_user}","gists_url":"https://api.github.com/users/awholegunch/gists{/gist_id}","starred_url":"https://api.github.com/users/awholegunch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awholegunch/subscriptions","organizations_url":"https://api.github.com/users/awholegunch/orgs","repos_url":"https://api.github.com/users/awholegunch/repos","events_url":"https://api.github.com/users/awholegunch/events{/privacy}","received_events_url":"https://api.github.com/users/awholegunch/received_events","type":"User","site_admin":false},"body":"These Adapter properties are optional:","created_at":"2015-01-28T23:17:27Z","updated_at":"2015-01-28T23:17:27Z","html_url":"https://github.com/caskdata/cdap/pull/1156#discussion_r23733601","pull_request_url":"https://api.github.com/repos/caskdata/cdap/pulls/1156","_links":{"self":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/comments/23733601"},"html":{"href":"https://github.com/caskdata/cdap/pull/1156#discussion_r23733601"},"pull_request":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/1156"}}},"pull_request":{"url":"https://api.github.com/repos/caskdata/cdap/pulls/1156","id":28233106,"html_url":"https://github.com/caskdata/cdap/pull/1156","diff_url":"https://github.com/caskdata/cdap/pull/1156.diff","patch_url":"https://github.com/caskdata/cdap/pull/1156.patch","issue_url":"https://api.github.com/repos/caskdata/cdap/issues/1156","number":1156,"state":"open","locked":false,"title":"Feature/cdap 1157 document stream conversion","user":{"login":"albertshau","id":5215004,"avatar_url":"https://avatars.githubusercontent.com/u/5215004?v=3","gravatar_id":"","url":"https://api.github.com/users/albertshau","html_url":"https://github.com/albertshau","followers_url":"https://api.github.com/users/albertshau/followers","following_url":"https://api.github.com/users/albertshau/following{/other_user}","gists_url":"https://api.github.com/users/albertshau/gists{/gist_id}","starred_url":"https://api.github.com/users/albertshau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/albertshau/subscriptions","organizations_url":"https://api.github.com/users/albertshau/orgs","repos_url":"https://api.github.com/users/albertshau/repos","events_url":"https://api.github.com/users/albertshau/events{/privacy}","received_events_url":"https://api.github.com/users/albertshau/received_events","type":"User","site_admin":false},"body":"","created_at":"2015-01-28T21:53:41Z","updated_at":"2015-01-28T23:17:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2cd414fb6a5374e906b7d8884005dc5a7539f117","assignee":{"login":"awholegunch","id":6394794,"avatar_url":"https://avatars.githubusercontent.com/u/6394794?v=3","gravatar_id":"","url":"https://api.github.com/users/awholegunch","html_url":"https://github.com/awholegunch","followers_url":"https://api.github.com/users/awholegunch/followers","following_url":"https://api.github.com/users/awholegunch/following{/other_user}","gists_url":"https://api.github.com/users/awholegunch/gists{/gist_id}","starred_url":"https://api.github.com/users/awholegunch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awholegunch/subscriptions","organizations_url":"https://api.github.com/users/awholegunch/orgs","repos_url":"https://api.github.com/users/awholegunch/repos","events_url":"https://api.github.com/users/awholegunch/events{/privacy}","received_events_url":"https://api.github.com/users/awholegunch/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/caskdata/cdap/pulls/1156/commits","review_comments_url":"https://api.github.com/repos/caskdata/cdap/pulls/1156/comments","review_comment_url":"https://api.github.com/repos/caskdata/cdap/pulls/comments/{number}","comments_url":"https://api.github.com/repos/caskdata/cdap/issues/1156/comments","statuses_url":"https://api.github.com/repos/caskdata/cdap/statuses/abaef5bee1a836b66215eca60a4649054cdf386c","head":{"label":"caskdata:feature/CDAP-1157-document-stream-conversion","ref":"feature/CDAP-1157-document-stream-conversion","sha":"abaef5bee1a836b66215eca60a4649054cdf386c","user":{"login":"caskdata","id":8123023,"avatar_url":"https://avatars.githubusercontent.com/u/8123023?v=3","gravatar_id":"","url":"https://api.github.com/users/caskdata","html_url":"https://github.com/caskdata","followers_url":"https://api.github.com/users/caskdata/followers","following_url":"https://api.github.com/users/caskdata/following{/other_user}","gists_url":"https://api.github.com/users/caskdata/gists{/gist_id}","starred_url":"https://api.github.com/users/caskdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caskdata/subscriptions","organizations_url":"https://api.github.com/users/caskdata/orgs","repos_url":"https://api.github.com/users/caskdata/repos","events_url":"https://api.github.com/users/caskdata/events{/privacy}","received_events_url":"https://api.github.com/users/caskdata/received_events","type":"Organization","site_admin":false},"repo":{"id":22542759,"name":"cdap","full_name":"caskdata/cdap","owner":{"login":"caskdata","id":8123023,"avatar_url":"https://avatars.githubusercontent.com/u/8123023?v=3","gravatar_id":"","url":"https://api.github.com/users/caskdata","html_url":"https://github.com/caskdata","followers_url":"https://api.github.com/users/caskdata/followers","following_url":"https://api.github.com/users/caskdata/following{/other_user}","gists_url":"https://api.github.com/users/caskdata/gists{/gist_id}","starred_url":"https://api.github.com/users/caskdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caskdata/subscriptions","organizations_url":"https://api.github.com/users/caskdata/orgs","repos_url":"https://api.github.com/users/caskdata/repos","events_url":"https://api.github.com/users/caskdata/events{/privacy}","received_events_url":"https://api.github.com/users/caskdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/caskdata/cdap","description":"Cask Data Application Platform (CDAP)","fork":false,"url":"https://api.github.com/repos/caskdata/cdap","forks_url":"https://api.github.com/repos/caskdata/cdap/forks","keys_url":"https://api.github.com/repos/caskdata/cdap/keys{/key_id}","collaborators_url":"https://api.github.com/repos/caskdata/cdap/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/caskdata/cdap/teams","hooks_url":"https://api.github.com/repos/caskdata/cdap/hooks","issue_events_url":"https://api.github.com/repos/caskdata/cdap/issues/events{/number}","events_url":"https://api.github.com/repos/caskdata/cdap/events","assignees_url":"https://api.github.com/repos/caskdata/cdap/assignees{/user}","branches_url":"https://api.github.com/repos/caskdata/cdap/branches{/branch}","tags_url":"https://api.github.com/repos/caskdata/cdap/tags","blobs_url":"https://api.github.com/repos/caskdata/cdap/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/caskdata/cdap/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/caskdata/cdap/git/refs{/sha}","trees_url":"https://api.github.com/repos/caskdata/cdap/git/trees{/sha}","statuses_url":"https://api.github.com/repos/caskdata/cdap/statuses/{sha}","languages_url":"https://api.github.com/repos/caskdata/cdap/languages","stargazers_url":"https://api.github.com/repos/caskdata/cdap/stargazers","contributors_url":"https://api.github.com/repos/caskdata/cdap/contributors","subscribers_url":"https://api.github.com/repos/caskdata/cdap/subscribers","subscription_url":"https://api.github.com/repos/caskdata/cdap/subscription","commits_url":"https://api.github.com/repos/caskdata/cdap/commits{/sha}","git_commits_url":"https://api.github.com/repos/caskdata/cdap/git/commits{/sha}","comments_url":"https://api.github.com/repos/caskdata/cdap/comments{/number}","issue_comment_url":"https://api.github.com/repos/caskdata/cdap/issues/comments/{number}","contents_url":"https://api.github.com/repos/caskdata/cdap/contents/{+path}","compare_url":"https://api.github.com/repos/caskdata/cdap/compare/{base}...{head}","merges_url":"https://api.github.com/repos/caskdata/cdap/merges","archive_url":"https://api.github.com/repos/caskdata/cdap/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/caskdata/cdap/downloads","issues_url":"https://api.github.com/repos/caskdata/cdap/issues{/number}","pulls_url":"https://api.github.com/repos/caskdata/cdap/pulls{/number}","milestones_url":"https://api.github.com/repos/caskdata/cdap/milestones{/number}","notifications_url":"https://api.github.com/repos/caskdata/cdap/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/caskdata/cdap/labels{/name}","releases_url":"https://api.github.com/repos/caskdata/cdap/releases{/id}","created_at":"2014-08-02T09:00:10Z","updated_at":"2015-01-28T04:55:43Z","pushed_at":"2015-01-28T23:10:26Z","git_url":"git://github.com/caskdata/cdap.git","ssh_url":"git@github.com:caskdata/cdap.git","clone_url":"https://github.com/caskdata/cdap.git","svn_url":"https://github.com/caskdata/cdap","homepage":"http://cdap.io","size":707764,"stargazers_count":23,"watchers_count":23,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":8,"mirror_url":null,"open_issues_count":13,"forks":8,"open_issues":13,"watchers":23,"default_branch":"develop"}},"base":{"label":"caskdata:release/2.7.0","ref":"release/2.7.0","sha":"61b5a7f714817df2864806ccf269b02fc541708b","user":{"login":"caskdata","id":8123023,"avatar_url":"https://avatars.githubusercontent.com/u/8123023?v=3","gravatar_id":"","url":"https://api.github.com/users/caskdata","html_url":"https://github.com/caskdata","followers_url":"https://api.github.com/users/caskdata/followers","following_url":"https://api.github.com/users/caskdata/following{/other_user}","gists_url":"https://api.github.com/users/caskdata/gists{/gist_id}","starred_url":"https://api.github.com/users/caskdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caskdata/subscriptions","organizations_url":"https://api.github.com/users/caskdata/orgs","repos_url":"https://api.github.com/users/caskdata/repos","events_url":"https://api.github.com/users/caskdata/events{/privacy}","received_events_url":"https://api.github.com/users/caskdata/received_events","type":"Organization","site_admin":false},"repo":{"id":22542759,"name":"cdap","full_name":"caskdata/cdap","owner":{"login":"caskdata","id":8123023,"avatar_url":"https://avatars.githubusercontent.com/u/8123023?v=3","gravatar_id":"","url":"https://api.github.com/users/caskdata","html_url":"https://github.com/caskdata","followers_url":"https://api.github.com/users/caskdata/followers","following_url":"https://api.github.com/users/caskdata/following{/other_user}","gists_url":"https://api.github.com/users/caskdata/gists{/gist_id}","starred_url":"https://api.github.com/users/caskdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caskdata/subscriptions","organizations_url":"https://api.github.com/users/caskdata/orgs","repos_url":"https://api.github.com/users/caskdata/repos","events_url":"https://api.github.com/users/caskdata/events{/privacy}","received_events_url":"https://api.github.com/users/caskdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/caskdata/cdap","description":"Cask Data Application Platform (CDAP)","fork":false,"url":"https://api.github.com/repos/caskdata/cdap","forks_url":"https://api.github.com/repos/caskdata/cdap/forks","keys_url":"https://api.github.com/repos/caskdata/cdap/keys{/key_id}","collaborators_url":"https://api.github.com/repos/caskdata/cdap/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/caskdata/cdap/teams","hooks_url":"https://api.github.com/repos/caskdata/cdap/hooks","issue_events_url":"https://api.github.com/repos/caskdata/cdap/issues/events{/number}","events_url":"https://api.github.com/repos/caskdata/cdap/events","assignees_url":"https://api.github.com/repos/caskdata/cdap/assignees{/user}","branches_url":"https://api.github.com/repos/caskdata/cdap/branches{/branch}","tags_url":"https://api.github.com/repos/caskdata/cdap/tags","blobs_url":"https://api.github.com/repos/caskdata/cdap/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/caskdata/cdap/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/caskdata/cdap/git/refs{/sha}","trees_url":"https://api.github.com/repos/caskdata/cdap/git/trees{/sha}","statuses_url":"https://api.github.com/repos/caskdata/cdap/statuses/{sha}","languages_url":"https://api.github.com/repos/caskdata/cdap/languages","stargazers_url":"https://api.github.com/repos/caskdata/cdap/stargazers","contributors_url":"https://api.github.com/repos/caskdata/cdap/contributors","subscribers_url":"https://api.github.com/repos/caskdata/cdap/subscribers","subscription_url":"https://api.github.com/repos/caskdata/cdap/subscription","commits_url":"https://api.github.com/repos/caskdata/cdap/commits{/sha}","git_commits_url":"https://api.github.com/repos/caskdata/cdap/git/commits{/sha}","comments_url":"https://api.github.com/repos/caskdata/cdap/comments{/number}","issue_comment_url":"https://api.github.com/repos/caskdata/cdap/issues/comments/{number}","contents_url":"https://api.github.com/repos/caskdata/cdap/contents/{+path}","compare_url":"https://api.github.com/repos/caskdata/cdap/compare/{base}...{head}","merges_url":"https://api.github.com/repos/caskdata/cdap/merges","archive_url":"https://api.github.com/repos/caskdata/cdap/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/caskdata/cdap/downloads","issues_url":"https://api.github.com/repos/caskdata/cdap/issues{/number}","pulls_url":"https://api.github.com/repos/caskdata/cdap/pulls{/number}","milestones_url":"https://api.github.com/repos/caskdata/cdap/milestones{/number}","notifications_url":"https://api.github.com/repos/caskdata/cdap/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/caskdata/cdap/labels{/name}","releases_url":"https://api.github.com/repos/caskdata/cdap/releases{/id}","created_at":"2014-08-02T09:00:10Z","updated_at":"2015-01-28T04:55:43Z","pushed_at":"2015-01-28T23:10:26Z","git_url":"git://github.com/caskdata/cdap.git","ssh_url":"git@github.com:caskdata/cdap.git","clone_url":"https://github.com/caskdata/cdap.git","svn_url":"https://github.com/caskdata/cdap","homepage":"http://cdap.io","size":707764,"stargazers_count":23,"watchers_count":23,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":8,"mirror_url":null,"open_issues_count":13,"forks":8,"open_issues":13,"watchers":23,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/1156"},"html":{"href":"https://github.com/caskdata/cdap/pull/1156"},"issue":{"href":"https://api.github.com/repos/caskdata/cdap/issues/1156"},"comments":{"href":"https://api.github.com/repos/caskdata/cdap/issues/1156/comments"},"review_comments":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/1156/comments"},"review_comment":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/caskdata/cdap/pulls/1156/commits"},"statuses":{"href":"https://api.github.com/repos/caskdata/cdap/statuses/abaef5bee1a836b66215eca60a4649054cdf386c"}}}}