{"action":"created","comment":{"url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/comments/25904941","id":25904941,"diff_hunk":"@@ -0,0 +1,108 @@\n+---\n+layout: page\n+title: Creating the Events Dataset\n+---\n+\n+This lesson shows you how to create a dataset suitable for storing standard event records. You define a dataset schema, a partition strategy, and a URI that specifies the storage scheme.\n+\n+## Defining the Schema\n+\n+The `standard_event.avsc` schema is self-describing, thanks to the _doc_ property for each of the fields. The fields store the `user_id` for the person who initiated the event, the user's IP address, and when the event occurred.\n+\n+### standard_event.avsc\n+\n+```JSON\n+{\n+  \"name\": \"StandardEvent\",\n+  \"namespace\": \"org.kitesdk.data.event\",\n+  \"type\": \"record\",\n+  \"doc\": \"A standard event type for logging, based on the paper 'The Unified Logging Infrastructure for Data Analytics at Twitter' by Lee et al, http://vldb.org/pvldb/vol5/p1771_georgelee_vldb2012.pdf\",\n+  \"fields\": [\n+    {\n+      \"name\": \"event_initiator\",\n+      \"type\": \"string\",\n+      \"doc\": \"Source of the event in the format {client,server}_{user,app}; for example, 'client_user'. Required.\"\n+    },\n+    {\n+      \"name\": \"event_name\",\n+      \"type\": \"string\",\n+      \"doc\": \"A hierarchical name for the event, with parts separated by ':'. Required.\"\n+    },\n+    {\n+      \"name\": \"user_id\",\n+      \"type\": \"long\",\n+      \"doc\": \"A unique identifier for the user. Required.\"\n+    },\n+    {\n+      \"name\": \"session_id\",\n+      \"type\": \"string\",\n+      \"doc\": \"A unique identifier for the session. Required.\"\n+    },\n+    {\n+      \"name\": \"ip\",\n+      \"type\": \"string\",\n+      \"doc\": \"The IP address of the host where the event originated. Required.\"\n+    },\n+    {\n+      \"name\": \"timestamp\",\n+      \"type\": \"long\",\n+      \"doc\": \"The point in time when the event occurred, represented as the number of milliseconds since January 1, 1970, 00:00:00 GMT. Required.\"\n+    }\n+  ]\n+}\n+```\n+\n+For convenience, save `standard_event.avsc` to the same directory where you installed the kite-dataset executable JAR.\n+\n+## Defining the Partition Strategy\n+\n+Analytics for the `events` dataset are time-based. Partitioning the dataset on the `timestamp` field allows Kite to go directly to the files for a particular day, ignoring files outside the chosen time period. Partition strategies are defined in JSON format. See [Partition Strategy JSON Format][partition-strategies].\n+\n+The following code sample defines a strategy that partitions a dataset by _year_, _month_, and _day_, based on a _timestamp_ field.\n+\n+### standard_event.json\n+\n+```\n+[ {\n+  \"source\" : \"timestamp\",\n+  \"type\" : \"year\",\n+  \"name\" : \"year\"\n+}, {\n+  \"source\" : \"timestamp\",\n+  \"type\" : \"month\",\n+  \"name\" : \"month\"\n+}, {\n+  \"source\" : \"timestamp\",\n+  \"type\" : \"day\",\n+  \"name\" : \"day\"\n+} ]\n+```\n+\n+For convenience, save `standard_event.json` to the same directory where you installed the `kite-dataset` executable JAR.\n+\n+[partition-strategies]:{{site.baseurl}}/Partition-Strategy-Format.html\n+\n+## Creating the Events Dataset Using the Kite CLI\n+\n+Create the _events_ dataset using the default Hive scheme.\n+\n+To create the _events_ dataset:\n+\n+1. Open a terminal window and navigate to the directory where you saved the schema file.\n+1. Use the `create` command to create the dataset.\n+\n+```\n+kite-dataset create events \\\n+             --schema standard_event.avsc \\\n+             --partition-by standard_event.json\n+```\n+\n+Use Hue to look at the schema and confirm that the dataset is ready to use.\n+\n+[http://quickstart.cloudera:8888/filebrowser/view//tmp/data/default/events/.metadata/schema.avsc](http://quickstart.cloudera:8888/filebrowser/view//tmp/data/default/events/.metadata/schema.avsc)\n+\n+## Next Steps\n+\n+You've created a dataset to store events captured as they happen. Now you can run a web application to create records in your new dataset. See [Capturing Events with Flume][capture-events].","path":"tutorials/create-events-dataset.md","position":106,"original_position":106,"commit_id":"16e423ce36afab49db4af82e087bea1d4a564ab8","original_commit_id":"16e423ce36afab49db4af82e087bea1d4a564ab8","user":{"login":"rdblue","id":87915,"avatar_url":"https://avatars.githubusercontent.com/u/87915?v=3","gravatar_id":"","url":"https://api.github.com/users/rdblue","html_url":"https://github.com/rdblue","followers_url":"https://api.github.com/users/rdblue/followers","following_url":"https://api.github.com/users/rdblue/following{/other_user}","gists_url":"https://api.github.com/users/rdblue/gists{/gist_id}","starred_url":"https://api.github.com/users/rdblue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rdblue/subscriptions","organizations_url":"https://api.github.com/users/rdblue/orgs","repos_url":"https://api.github.com/users/rdblue/repos","events_url":"https://api.github.com/users/rdblue/events{/privacy}","received_events_url":"https://api.github.com/users/rdblue/received_events","type":"User","site_admin":false},"body":"I think the context given in the first sentence should be in the setup for this tutorial. The tutorial series creates a dataset to load with flume as events come in, but we actually use it in a few ways: we bulk-load fake data, stream events in through flume, and read events with Crunch. Saying that the dataset was created to do one of those is a little misleading and leaves the me wondering which part makes this dataset specific to storing events as they happen.","created_at":"2015-03-05T21:47:50Z","updated_at":"2015-03-05T21:47:50Z","html_url":"https://github.com/kite-sdk/kite-docs/pull/79#discussion_r25904941","pull_request_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/79","_links":{"self":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/comments/25904941"},"html":{"href":"https://github.com/kite-sdk/kite-docs/pull/79#discussion_r25904941"},"pull_request":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/79"}}},"pull_request":{"url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/79","id":30514967,"html_url":"https://github.com/kite-sdk/kite-docs/pull/79","diff_url":"https://github.com/kite-sdk/kite-docs/pull/79.diff","patch_url":"https://github.com/kite-sdk/kite-docs/pull/79.patch","issue_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/79","number":79,"state":"open","locked":false,"title":"CDK-29, CDK-466, CDK-827: Include use of CLI flume-config.","user":{"login":"DennisDawson","id":6788473,"avatar_url":"https://avatars.githubusercontent.com/u/6788473?v=3","gravatar_id":"","url":"https://api.github.com/users/DennisDawson","html_url":"https://github.com/DennisDawson","followers_url":"https://api.github.com/users/DennisDawson/followers","following_url":"https://api.github.com/users/DennisDawson/following{/other_user}","gists_url":"https://api.github.com/users/DennisDawson/gists{/gist_id}","starred_url":"https://api.github.com/users/DennisDawson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DennisDawson/subscriptions","organizations_url":"https://api.github.com/users/DennisDawson/orgs","repos_url":"https://api.github.com/users/DennisDawson/repos","events_url":"https://api.github.com/users/DennisDawson/events{/privacy}","received_events_url":"https://api.github.com/users/DennisDawson/received_events","type":"User","site_admin":false},"body":"","created_at":"2015-03-04T23:15:10Z","updated_at":"2015-03-05T21:47:50Z","closed_at":null,"merged_at":null,"merge_commit_sha":"38aae47e86b3927aaa002a4293ad9f27a23cda52","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/79/commits","review_comments_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/79/comments","review_comment_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/79/comments","statuses_url":"https://api.github.com/repos/kite-sdk/kite-docs/statuses/16e423ce36afab49db4af82e087bea1d4a564ab8","head":{"label":"kite-sdk:addGenerateEventsTutorial","ref":"addGenerateEventsTutorial","sha":"16e423ce36afab49db4af82e087bea1d4a564ab8","user":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"repo":{"id":21282660,"name":"kite-docs","full_name":"kite-sdk/kite-docs","owner":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kite-sdk/kite-docs","description":"Kite documentation and tutorials","fork":false,"url":"https://api.github.com/repos/kite-sdk/kite-docs","forks_url":"https://api.github.com/repos/kite-sdk/kite-docs/forks","keys_url":"https://api.github.com/repos/kite-sdk/kite-docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kite-sdk/kite-docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kite-sdk/kite-docs/teams","hooks_url":"https://api.github.com/repos/kite-sdk/kite-docs/hooks","issue_events_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/events{/number}","events_url":"https://api.github.com/repos/kite-sdk/kite-docs/events","assignees_url":"https://api.github.com/repos/kite-sdk/kite-docs/assignees{/user}","branches_url":"https://api.github.com/repos/kite-sdk/kite-docs/branches{/branch}","tags_url":"https://api.github.com/repos/kite-sdk/kite-docs/tags","blobs_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kite-sdk/kite-docs/statuses/{sha}","languages_url":"https://api.github.com/repos/kite-sdk/kite-docs/languages","stargazers_url":"https://api.github.com/repos/kite-sdk/kite-docs/stargazers","contributors_url":"https://api.github.com/repos/kite-sdk/kite-docs/contributors","subscribers_url":"https://api.github.com/repos/kite-sdk/kite-docs/subscribers","subscription_url":"https://api.github.com/repos/kite-sdk/kite-docs/subscription","commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/kite-sdk/kite-docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/kite-sdk/kite-docs/contents/{+path}","compare_url":"https://api.github.com/repos/kite-sdk/kite-docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kite-sdk/kite-docs/merges","archive_url":"https://api.github.com/repos/kite-sdk/kite-docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kite-sdk/kite-docs/downloads","issues_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues{/number}","pulls_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls{/number}","milestones_url":"https://api.github.com/repos/kite-sdk/kite-docs/milestones{/number}","notifications_url":"https://api.github.com/repos/kite-sdk/kite-docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kite-sdk/kite-docs/labels{/name}","releases_url":"https://api.github.com/repos/kite-sdk/kite-docs/releases{/id}","created_at":"2014-06-27T16:54:57Z","updated_at":"2015-02-25T01:01:47Z","pushed_at":"2015-03-05T18:52:59Z","git_url":"git://github.com/kite-sdk/kite-docs.git","ssh_url":"git@github.com:kite-sdk/kite-docs.git","clone_url":"https://github.com/kite-sdk/kite-docs.git","svn_url":"https://github.com/kite-sdk/kite-docs","homepage":null,"size":2308,"stargazers_count":1,"watchers_count":1,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"open_issues_count":2,"forks":8,"open_issues":2,"watchers":1,"default_branch":"master"}},"base":{"label":"kite-sdk:master","ref":"master","sha":"137b9bc7e123f50c668e9a4dd76e92b8ecebbb6f","user":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"repo":{"id":21282660,"name":"kite-docs","full_name":"kite-sdk/kite-docs","owner":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kite-sdk/kite-docs","description":"Kite documentation and tutorials","fork":false,"url":"https://api.github.com/repos/kite-sdk/kite-docs","forks_url":"https://api.github.com/repos/kite-sdk/kite-docs/forks","keys_url":"https://api.github.com/repos/kite-sdk/kite-docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kite-sdk/kite-docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kite-sdk/kite-docs/teams","hooks_url":"https://api.github.com/repos/kite-sdk/kite-docs/hooks","issue_events_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/events{/number}","events_url":"https://api.github.com/repos/kite-sdk/kite-docs/events","assignees_url":"https://api.github.com/repos/kite-sdk/kite-docs/assignees{/user}","branches_url":"https://api.github.com/repos/kite-sdk/kite-docs/branches{/branch}","tags_url":"https://api.github.com/repos/kite-sdk/kite-docs/tags","blobs_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kite-sdk/kite-docs/statuses/{sha}","languages_url":"https://api.github.com/repos/kite-sdk/kite-docs/languages","stargazers_url":"https://api.github.com/repos/kite-sdk/kite-docs/stargazers","contributors_url":"https://api.github.com/repos/kite-sdk/kite-docs/contributors","subscribers_url":"https://api.github.com/repos/kite-sdk/kite-docs/subscribers","subscription_url":"https://api.github.com/repos/kite-sdk/kite-docs/subscription","commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/kite-sdk/kite-docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/kite-sdk/kite-docs/contents/{+path}","compare_url":"https://api.github.com/repos/kite-sdk/kite-docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kite-sdk/kite-docs/merges","archive_url":"https://api.github.com/repos/kite-sdk/kite-docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kite-sdk/kite-docs/downloads","issues_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues{/number}","pulls_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls{/number}","milestones_url":"https://api.github.com/repos/kite-sdk/kite-docs/milestones{/number}","notifications_url":"https://api.github.com/repos/kite-sdk/kite-docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kite-sdk/kite-docs/labels{/name}","releases_url":"https://api.github.com/repos/kite-sdk/kite-docs/releases{/id}","created_at":"2014-06-27T16:54:57Z","updated_at":"2015-02-25T01:01:47Z","pushed_at":"2015-03-05T18:52:59Z","git_url":"git://github.com/kite-sdk/kite-docs.git","ssh_url":"git@github.com:kite-sdk/kite-docs.git","clone_url":"https://github.com/kite-sdk/kite-docs.git","svn_url":"https://github.com/kite-sdk/kite-docs","homepage":null,"size":2308,"stargazers_count":1,"watchers_count":1,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"open_issues_count":2,"forks":8,"open_issues":2,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/79"},"html":{"href":"https://github.com/kite-sdk/kite-docs/pull/79"},"issue":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/issues/79"},"comments":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/issues/79/comments"},"review_comments":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/79/comments"},"review_comment":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/79/commits"},"statuses":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/statuses/16e423ce36afab49db4af82e087bea1d4a564ab8"}}}}