{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7770","labels_url":"https://api.github.com/repos/apache/spark/issues/7770/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7770/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7770/events","html_url":"https://github.com/apache/spark/pull/7770","id":98073908,"number":7770,"title":"[SPARK-8735] [SQL] Expose memory usage for shuffles, joins and aggregations","user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":68,"created_at":"2015-07-30T02:47:08Z","updated_at":"2015-08-02T22:38:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7770","html_url":"https://github.com/apache/spark/pull/7770","diff_url":"https://github.com/apache/spark/pull/7770.diff","patch_url":"https://github.com/apache/spark/pull/7770.patch"},"body":"This patch exposes the memory used by internal data structures on the SparkUI. This tracks memory used by all spilling operations and SQL operators backed by Tungsten, e.g. `BroadcastHashJoin`, `ExternalSort`, `GeneratedAggregate` etc. The metric exposed is \"peak execution memory\", which broadly refers to the peak in-memory sizes of each of these data structure.\r\n\r\nA separate patch will extend this by linking the new information to the SQL operators themselves.\r\n\r\n<img width=\"950\" alt=\"screen shot 2015-07-29 at 7 43 17 pm\" src=\"https://cloud.githubusercontent.com/assets/2133137/8974776/b90fc980-362a-11e5-9e2b-842da75b1641.png\">\r\n<img width=\"802\" alt=\"screen shot 2015-07-29 at 7 43 05 pm\" src=\"https://cloud.githubusercontent.com/assets/2133137/8974777/baa76492-362a-11e5-9b77-e364a6a6b64e.png\">\n\n<!-- Reviewable:start -->\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/7770)\n<!-- Reviewable:end -->\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/127077823","html_url":"https://github.com/apache/spark/pull/7770#issuecomment-127077823","issue_url":"https://api.github.com/repos/apache/spark/issues/7770","id":127077823,"user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"created_at":"2015-08-02T22:38:17Z","updated_at":"2015-08-02T22:38:17Z","body":"It seems the main source of outstanding debate is whether to display this metric only when unsafe is enabled, which is the proposal of this patch.\r\n\r\nThe motivation is as follows: there are SQL operators whose memory usage is both high and difficult to track without sacrificing performance (e.g. `Sort` simply uses `Arrays.sort`, or `BroadcastHashJoin` with `GeneralHashedRelation` uses a Java hash map under the hood). It would be misleading if we say the memory usage in these operators is low simply because we do not track them.\r\n\r\nThere are two ways of fixing this: (1) only display this when unsafe is enabled, and (2) detect cases when jobs are triggered due to the operators we don't track. (2) is of a significantly wider scope than the intention of this patch, so we just went with (1) here. Note that this means we don't expose this metric even for simple non-SQL aggregations (e.g. `reduceByKey`), but working around that would require us to detect whether a job is a SQL job, which is being implemented in #7774 but outside the scope of this individual patch."}}