{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/26088634","id":26088634,"diff_hunk":"@@ -1801,40 +1894,40 @@ temporary data rate increases maybe fine as long as the delay reduces back to a\n \n ## Memory Tuning\n Tuning the memory usage and GC behavior of Spark applications have been discussed in great detail\n-in the [Tuning Guide](tuning.html). It is recommended that you read that. In this section,\n-we highlight a few customizations that are strongly recommended to minimize GC related pauses\n-in Spark Streaming applications and achieving more consistent batch processing times.\n-\n-* **Default persistence level of DStreams**: Unlike RDDs, the default persistence level of DStreams\n-serializes the data in memory (that is,\n-[StorageLevel.MEMORY_ONLY_SER](api/scala/index.html#org.apache.spark.storage.StorageLevel$) for\n-DStream compared to\n-[StorageLevel.MEMORY_ONLY](api/scala/index.html#org.apache.spark.storage.StorageLevel$) for RDDs).\n-Even though keeping the data serialized incurs higher serialization/deserialization overheads,\n-it significantly reduces GC pauses.\n-\n-* **Clearing persistent RDDs**: By default, all persistent RDDs generated by Spark Streaming will\n- be cleared from memory based on Spark's built-in policy (LRU). If `spark.cleaner.ttl` is set,\n- then persistent RDDs that are older than that value are periodically cleared. As mentioned\n- [earlier](#operation), this needs to be careful set based on operations used in the Spark\n- Streaming program. However, a smarter unpersisting of RDDs can be enabled by setting the\n- [configuration property](configuration.html#spark-properties) `spark.streaming.unpersist` to\n- `true`. This makes the system to figure out which RDDs are not necessary to be kept around and\n- unpersists them. This is likely to reduce\n- the RDD memory usage of Spark, potentially improving GC behavior as well.\n-\n-* **Concurrent garbage collector**: Using the concurrent mark-and-sweep GC further\n-minimizes the variability of GC pauses. Even though concurrent GC is known to reduce the\n+in the [Tuning Guide](tuning.html#memory-tuning). It is strongly recommended that you read that. In this section, we discuss a few tuning parameters specifically in the context of Spark Streaming applications.\n+\n+The amount of cluster memory required by a Spark Streaming application depends heavily on the type of transformations used. For example, if you want to use a window operation on last 10 minutes of data, then your cluster should have sufficient memory to hold 10 minutes of worth of data in memory. Or if you want to use `updateStateByKey` with a large number of keys, then the necessary memory  will be high. On the contrary, if you want to do a simple map-filter-store operation, then necessary memory will be low.\n+\n+In general, since the data received through receivers are stored with StorageLevel.MEMORY_AND_DISK_SER_2, the data that does not fit in memory will spill over to the disk. This may reduce the performance of the streaming application, and hence it is advised to provide sufficient memory as required by your streaming application. Its best to try and see the memory usage on a small scale and estimate accordingly. \n+\n+Another aspect of memory tuning is garbage collection. For a streaming application that require low latency, it is undesirable to have large pauses caused by JVM Garbage Collection. \n+\n+There are a few parameters that can help you tune the memory usage and GC overheads.\n+\n+* **Persistence Level of DStreams**: As mentioned earlier in the [Data Serialization](#data-serialization) section, the input data and RDDs are by default persisted as serialized bytes. This reduces both, the memory usage and GC overheads, compared to deserialized persistence. Enabling Kryo serialization, further reduces serialized sizes and memory usage. Further reduction in usage using compression (see the Spark configuration `spark.rdd.compress`) can come at the cost of CPU time.\n+\n+* **Clearing old data**: By default, all input data and persisted RDDs generated by DStream transformations are automatically cleared. Spark Streaming decides when to clear the data based on the transformations that are used. For example, if you are using window operation of 10 minutes, then Spark Streaming will keep around last 10 minutes of data, and actively throw away older data. \n+Data can be retained for longer duration (e.g. interactively querying older data) by setting `streamingContext.remember`.\n+\n+* **CMS Garbage Collector**: Use of the concurrent mark-and-sweep GC is strongly recommended for keeping GC-related pauses consistently low. Even though concurrent GC is known to reduce the\n overall processing throughput of the system, its use is still recommended to achieve more\n-consistent batch processing times.\n+consistent batch processing times. Make sure you set the CMS GC on both the driver (using `--driver-java-options` in `spark-submit`) and the executors (using [Spark configuration](configuration.html#runtime-environment) `spark.executor.extraJavaOptions`).\n+\n+* **Other tips**: To further reduce GC overheads, here are some more tips to try.\n+    - Use Tachyon for off-heap storage of persisted RDDs. See more detail in the [Spark Programming Guide](programming-guide.html#rdd-persistence).\n+    - Use more executors with smaller heap sizes. This will reduce the GC pressue within each JVM heap.","path":"docs/streaming-programming-guide.md","position":288,"original_position":288,"commit_id":"86c4c2ad23d1aa02d995361211870f12ff812caa","original_commit_id":"86c4c2ad23d1aa02d995361211870f12ff812caa","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"body":"Typo: pressue -> pressure (missing an 'r').","created_at":"2015-03-09T23:58:18Z","updated_at":"2015-03-09T23:58:18Z","html_url":"https://github.com/apache/spark/pull/4956#discussion_r26088634","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/4956","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/26088634"},"html":{"href":"https://github.com/apache/spark/pull/4956#discussion_r26088634"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/4956"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4956","id":30808553,"html_url":"https://github.com/apache/spark/pull/4956","diff_url":"https://github.com/apache/spark/pull/4956.diff","patch_url":"https://github.com/apache/spark/pull/4956.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/4956","number":4956,"state":"open","locked":false,"title":"[SPARK-6128][Streaming][Documentation] Updates to Spark Streaming Programming Guide","user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"body":"Updates to the documentation are as follows: \r\n\r\n- Added information on Kafka Direct API and Kafka Python API\r\n- Added joins to the main streaming guide\r\n- Improved details on the fault-tolerance semantics\r\n\r\nGenerated docs located here\r\nhttp://people.apache.org/~tdas/spark-1.3.0-temp-docs/streaming-programming-guide.html#fault-tolerance-semantics","created_at":"2015-03-09T23:00:21Z","updated_at":"2015-03-09T23:58:18Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2c4dfb4f5f00a5e308d2d42807deece7b9572272","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/4956/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/4956/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/4956/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/86c4c2ad23d1aa02d995361211870f12ff812caa","head":{"label":"tdas:streaming-guide-update-1.3","ref":"streaming-guide-update-1.3","sha":"86c4c2ad23d1aa02d995361211870f12ff812caa","user":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"repo":{"id":17603343,"name":"spark","full_name":"tdas/spark","owner":{"login":"tdas","id":663212,"avatar_url":"https://avatars.githubusercontent.com/u/663212?v=3","gravatar_id":"","url":"https://api.github.com/users/tdas","html_url":"https://github.com/tdas","followers_url":"https://api.github.com/users/tdas/followers","following_url":"https://api.github.com/users/tdas/following{/other_user}","gists_url":"https://api.github.com/users/tdas/gists{/gist_id}","starred_url":"https://api.github.com/users/tdas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdas/subscriptions","organizations_url":"https://api.github.com/users/tdas/orgs","repos_url":"https://api.github.com/users/tdas/repos","events_url":"https://api.github.com/users/tdas/events{/privacy}","received_events_url":"https://api.github.com/users/tdas/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tdas/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/tdas/spark","forks_url":"https://api.github.com/repos/tdas/spark/forks","keys_url":"https://api.github.com/repos/tdas/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tdas/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tdas/spark/teams","hooks_url":"https://api.github.com/repos/tdas/spark/hooks","issue_events_url":"https://api.github.com/repos/tdas/spark/issues/events{/number}","events_url":"https://api.github.com/repos/tdas/spark/events","assignees_url":"https://api.github.com/repos/tdas/spark/assignees{/user}","branches_url":"https://api.github.com/repos/tdas/spark/branches{/branch}","tags_url":"https://api.github.com/repos/tdas/spark/tags","blobs_url":"https://api.github.com/repos/tdas/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tdas/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tdas/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/tdas/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tdas/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/tdas/spark/languages","stargazers_url":"https://api.github.com/repos/tdas/spark/stargazers","contributors_url":"https://api.github.com/repos/tdas/spark/contributors","subscribers_url":"https://api.github.com/repos/tdas/spark/subscribers","subscription_url":"https://api.github.com/repos/tdas/spark/subscription","commits_url":"https://api.github.com/repos/tdas/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/tdas/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/tdas/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/tdas/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/tdas/spark/contents/{+path}","compare_url":"https://api.github.com/repos/tdas/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tdas/spark/merges","archive_url":"https://api.github.com/repos/tdas/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tdas/spark/downloads","issues_url":"https://api.github.com/repos/tdas/spark/issues{/number}","pulls_url":"https://api.github.com/repos/tdas/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/tdas/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/tdas/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tdas/spark/labels{/name}","releases_url":"https://api.github.com/repos/tdas/spark/releases{/id}","created_at":"2014-03-10T18:19:48Z","updated_at":"2014-10-09T07:43:39Z","pushed_at":"2015-03-09T22:50:21Z","git_url":"git://github.com/tdas/spark.git","ssh_url":"git@github.com:tdas/spark.git","clone_url":"https://github.com/tdas/spark.git","svn_url":"https://github.com/tdas/spark","homepage":null,"size":94385,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":1,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"70f88148bb04161a1a4968230d8e3fc7e3f8321a","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-03-09T23:24:28Z","pushed_at":"2015-03-09T23:24:28Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1356185,"stargazers_count":3122,"watchers_count":3122,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2707,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":280,"forks":2707,"open_issues":280,"watchers":3122,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/4956"},"html":{"href":"https://github.com/apache/spark/pull/4956"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/4956"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/4956/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/4956/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/4956/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/86c4c2ad23d1aa02d995361211870f12ff812caa"}}}}