{"action":"created","comment":{"url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/comments/23965776","id":23965776,"diff_hunk":"@@ -0,0 +1,46 @@\n+---\n+layout: page\n+title: Using Kite Properties\n+---\n+\n+Kite gives you the option of changing settings of existing system properties to enhance performance. You also have the option of creating custom properties for additional control over how Kite reads or writes your data.\n+\n+## Setting Kite Properties\n+\n+There are several dataset properties that affect how files are written. These settings are especially relevant for Parquet because it buffers records in memory. \n+\n+You can set properties on your datasets with [`DatasetDescriptor.Builder.property`][dataset-descriptor-builder] when using the API, or with the `--set` option from the command line.\n+\n+### kite.write.cache-size\n+\n+`kite.write.cache-size` controls the number of files kept open by an HDFS or Hive dataset writer. The default cache size is 10.\n+\n+Writers open one file per partition to which they write records. When the writer receives a record that goes in a new partition (one for which there isn't an open file) it creates a new file in that partition. If the number of open files exceeds the cache size, Kite closes the file that was used least recently.\n+\n+For example, you might have an application writing a year's worth of data to year/month partitions. There are 12 partitions, but only 10 files open at the same time. The application must constantly close and open files, which slows down your writes. If you increase the writer cache to 12, you can have files for all 12 months open at once.\n+\n+### parquet.block.size\n+\n+Kite passes descriptor properties to the underlying file formats. For example, Parquet defines `parquet.block.size`, which is approximately the amount of data that is buffered before writing a group of records (a \"row group\"). `parquet.block.size` defaults to 128MB.\n+\n+#### Avoiding Parquet OutOfMemory Exceptions\n+\n+The amount of data kept in memory for each file could be up to the Parquet block size in bytes. That means that the upper bound for a writer's memory consumption is `parquet.block.size` multiplied by the `kite.writer.cache-size`. It is important that this number doesn't exceed a reasonable portion of the heap memory allocated to the process, or else the write could fail with an `OutOfMemoryException`. \n+\n+```\n+kite-dataset update <uri> --set kite.writer.cache-size=2","path":"using-kite-properties.md","position":31,"original_position":31,"commit_id":"ed8e04dabd8e3df5181eac88b63c12cddf2026d1","original_commit_id":"ed8e04dabd8e3df5181eac88b63c12cddf2026d1","user":{"login":"rdblue","id":87915,"avatar_url":"https://avatars.githubusercontent.com/u/87915?v=3","gravatar_id":"","url":"https://api.github.com/users/rdblue","html_url":"https://github.com/rdblue","followers_url":"https://api.github.com/users/rdblue/followers","following_url":"https://api.github.com/users/rdblue/following{/other_user}","gists_url":"https://api.github.com/users/rdblue/gists{/gist_id}","starred_url":"https://api.github.com/users/rdblue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rdblue/subscriptions","organizations_url":"https://api.github.com/users/rdblue/orgs","repos_url":"https://api.github.com/users/rdblue/repos","events_url":"https://api.github.com/users/rdblue/events{/privacy}","received_events_url":"https://api.github.com/users/rdblue/received_events","type":"User","site_admin":false},"body":"This command needs more context.","created_at":"2015-02-02T22:37:10Z","updated_at":"2015-02-02T22:37:10Z","html_url":"https://github.com/kite-sdk/kite-docs/pull/70#discussion_r23965776","pull_request_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/70","_links":{"self":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/comments/23965776"},"html":{"href":"https://github.com/kite-sdk/kite-docs/pull/70#discussion_r23965776"},"pull_request":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/70"}}},"pull_request":{"url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/70","id":27057363,"html_url":"https://github.com/kite-sdk/kite-docs/pull/70","diff_url":"https://github.com/kite-sdk/kite-docs/pull/70.diff","patch_url":"https://github.com/kite-sdk/kite-docs/pull/70.patch","issue_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/70","number":70,"state":"open","locked":false,"title":"CDK-865,  CDK-846 : New topics based on old kite-data-module-overview.","user":{"login":"DennisDawson","id":6788473,"avatar_url":"https://avatars.githubusercontent.com/u/6788473?v=3","gravatar_id":"","url":"https://api.github.com/users/DennisDawson","html_url":"https://github.com/DennisDawson","followers_url":"https://api.github.com/users/DennisDawson/followers","following_url":"https://api.github.com/users/DennisDawson/following{/other_user}","gists_url":"https://api.github.com/users/DennisDawson/gists{/gist_id}","starred_url":"https://api.github.com/users/DennisDawson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DennisDawson/subscriptions","organizations_url":"https://api.github.com/users/DennisDawson/orgs","repos_url":"https://api.github.com/users/DennisDawson/repos","events_url":"https://api.github.com/users/DennisDawson/events{/privacy}","received_events_url":"https://api.github.com/users/DennisDawson/received_events","type":"User","site_admin":false},"body":"","created_at":"2015-01-08T20:32:10Z","updated_at":"2015-02-02T22:37:10Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9a2f43e3a40045c3fc359d90163f3deaaeb075c1","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/70/commits","review_comments_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/70/comments","review_comment_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/comments/{number}","comments_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/70/comments","statuses_url":"https://api.github.com/repos/kite-sdk/kite-docs/statuses/ed8e04dabd8e3df5181eac88b63c12cddf2026d1","head":{"label":"kite-sdk:addIntroDatasets","ref":"addIntroDatasets","sha":"ed8e04dabd8e3df5181eac88b63c12cddf2026d1","user":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"repo":{"id":21282660,"name":"kite-docs","full_name":"kite-sdk/kite-docs","owner":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kite-sdk/kite-docs","description":"Kite documentation and tutorials","fork":false,"url":"https://api.github.com/repos/kite-sdk/kite-docs","forks_url":"https://api.github.com/repos/kite-sdk/kite-docs/forks","keys_url":"https://api.github.com/repos/kite-sdk/kite-docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kite-sdk/kite-docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kite-sdk/kite-docs/teams","hooks_url":"https://api.github.com/repos/kite-sdk/kite-docs/hooks","issue_events_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/events{/number}","events_url":"https://api.github.com/repos/kite-sdk/kite-docs/events","assignees_url":"https://api.github.com/repos/kite-sdk/kite-docs/assignees{/user}","branches_url":"https://api.github.com/repos/kite-sdk/kite-docs/branches{/branch}","tags_url":"https://api.github.com/repos/kite-sdk/kite-docs/tags","blobs_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kite-sdk/kite-docs/statuses/{sha}","languages_url":"https://api.github.com/repos/kite-sdk/kite-docs/languages","stargazers_url":"https://api.github.com/repos/kite-sdk/kite-docs/stargazers","contributors_url":"https://api.github.com/repos/kite-sdk/kite-docs/contributors","subscribers_url":"https://api.github.com/repos/kite-sdk/kite-docs/subscribers","subscription_url":"https://api.github.com/repos/kite-sdk/kite-docs/subscription","commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/kite-sdk/kite-docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/comments/{number}","contents_url":"https://api.github.com/repos/kite-sdk/kite-docs/contents/{+path}","compare_url":"https://api.github.com/repos/kite-sdk/kite-docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kite-sdk/kite-docs/merges","archive_url":"https://api.github.com/repos/kite-sdk/kite-docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kite-sdk/kite-docs/downloads","issues_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues{/number}","pulls_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls{/number}","milestones_url":"https://api.github.com/repos/kite-sdk/kite-docs/milestones{/number}","notifications_url":"https://api.github.com/repos/kite-sdk/kite-docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kite-sdk/kite-docs/labels{/name}","releases_url":"https://api.github.com/repos/kite-sdk/kite-docs/releases{/id}","created_at":"2014-06-27T16:54:57Z","updated_at":"2015-01-31T03:24:08Z","pushed_at":"2015-02-02T19:17:58Z","git_url":"git://github.com/kite-sdk/kite-docs.git","ssh_url":"git@github.com:kite-sdk/kite-docs.git","clone_url":"https://github.com/kite-sdk/kite-docs.git","svn_url":"https://github.com/kite-sdk/kite-docs","homepage":null,"size":2003,"stargazers_count":1,"watchers_count":1,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":7,"mirror_url":null,"open_issues_count":3,"forks":7,"open_issues":3,"watchers":1,"default_branch":"master"}},"base":{"label":"kite-sdk:master","ref":"master","sha":"f3273dc00e3bcb5ecf2c51604a8e3b2a071f8f04","user":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"repo":{"id":21282660,"name":"kite-docs","full_name":"kite-sdk/kite-docs","owner":{"login":"kite-sdk","id":5732728,"avatar_url":"https://avatars.githubusercontent.com/u/5732728?v=3","gravatar_id":"","url":"https://api.github.com/users/kite-sdk","html_url":"https://github.com/kite-sdk","followers_url":"https://api.github.com/users/kite-sdk/followers","following_url":"https://api.github.com/users/kite-sdk/following{/other_user}","gists_url":"https://api.github.com/users/kite-sdk/gists{/gist_id}","starred_url":"https://api.github.com/users/kite-sdk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kite-sdk/subscriptions","organizations_url":"https://api.github.com/users/kite-sdk/orgs","repos_url":"https://api.github.com/users/kite-sdk/repos","events_url":"https://api.github.com/users/kite-sdk/events{/privacy}","received_events_url":"https://api.github.com/users/kite-sdk/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kite-sdk/kite-docs","description":"Kite documentation and tutorials","fork":false,"url":"https://api.github.com/repos/kite-sdk/kite-docs","forks_url":"https://api.github.com/repos/kite-sdk/kite-docs/forks","keys_url":"https://api.github.com/repos/kite-sdk/kite-docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kite-sdk/kite-docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kite-sdk/kite-docs/teams","hooks_url":"https://api.github.com/repos/kite-sdk/kite-docs/hooks","issue_events_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/events{/number}","events_url":"https://api.github.com/repos/kite-sdk/kite-docs/events","assignees_url":"https://api.github.com/repos/kite-sdk/kite-docs/assignees{/user}","branches_url":"https://api.github.com/repos/kite-sdk/kite-docs/branches{/branch}","tags_url":"https://api.github.com/repos/kite-sdk/kite-docs/tags","blobs_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kite-sdk/kite-docs/statuses/{sha}","languages_url":"https://api.github.com/repos/kite-sdk/kite-docs/languages","stargazers_url":"https://api.github.com/repos/kite-sdk/kite-docs/stargazers","contributors_url":"https://api.github.com/repos/kite-sdk/kite-docs/contributors","subscribers_url":"https://api.github.com/repos/kite-sdk/kite-docs/subscribers","subscription_url":"https://api.github.com/repos/kite-sdk/kite-docs/subscription","commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/kite-sdk/kite-docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/kite-sdk/kite-docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues/comments/{number}","contents_url":"https://api.github.com/repos/kite-sdk/kite-docs/contents/{+path}","compare_url":"https://api.github.com/repos/kite-sdk/kite-docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kite-sdk/kite-docs/merges","archive_url":"https://api.github.com/repos/kite-sdk/kite-docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kite-sdk/kite-docs/downloads","issues_url":"https://api.github.com/repos/kite-sdk/kite-docs/issues{/number}","pulls_url":"https://api.github.com/repos/kite-sdk/kite-docs/pulls{/number}","milestones_url":"https://api.github.com/repos/kite-sdk/kite-docs/milestones{/number}","notifications_url":"https://api.github.com/repos/kite-sdk/kite-docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kite-sdk/kite-docs/labels{/name}","releases_url":"https://api.github.com/repos/kite-sdk/kite-docs/releases{/id}","created_at":"2014-06-27T16:54:57Z","updated_at":"2015-01-31T03:24:08Z","pushed_at":"2015-02-02T19:17:58Z","git_url":"git://github.com/kite-sdk/kite-docs.git","ssh_url":"git@github.com:kite-sdk/kite-docs.git","clone_url":"https://github.com/kite-sdk/kite-docs.git","svn_url":"https://github.com/kite-sdk/kite-docs","homepage":null,"size":2003,"stargazers_count":1,"watchers_count":1,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":7,"mirror_url":null,"open_issues_count":3,"forks":7,"open_issues":3,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/70"},"html":{"href":"https://github.com/kite-sdk/kite-docs/pull/70"},"issue":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/issues/70"},"comments":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/issues/70/comments"},"review_comments":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/70/comments"},"review_comment":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/pulls/70/commits"},"statuses":{"href":"https://api.github.com/repos/kite-sdk/kite-docs/statuses/ed8e04dabd8e3df5181eac88b63c12cddf2026d1"}}}}