{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7753","labels_url":"https://api.github.com/repos/apache/spark/issues/7753/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7753/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7753/events","html_url":"https://github.com/apache/spark/pull/7753","id":97991340,"number":7753,"title":"[SPARK-9104][SPARK-9105][SPARK-9106][SPARK-9107][CORE][WIP] Netty network layer memory usage on webUI","user":{"login":"liyezhang556520","id":4716022,"avatar_url":"https://avatars.githubusercontent.com/u/4716022?v=3","gravatar_id":"","url":"https://api.github.com/users/liyezhang556520","html_url":"https://github.com/liyezhang556520","followers_url":"https://api.github.com/users/liyezhang556520/followers","following_url":"https://api.github.com/users/liyezhang556520/following{/other_user}","gists_url":"https://api.github.com/users/liyezhang556520/gists{/gist_id}","starred_url":"https://api.github.com/users/liyezhang556520/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liyezhang556520/subscriptions","organizations_url":"https://api.github.com/users/liyezhang556520/orgs","repos_url":"https://api.github.com/users/liyezhang556520/repos","events_url":"https://api.github.com/users/liyezhang556520/events{/privacy}","received_events_url":"https://api.github.com/users/liyezhang556520/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":26,"created_at":"2015-07-29T17:49:24Z","updated_at":"2015-11-03T21:12:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7753","html_url":"https://github.com/apache/spark/pull/7753","diff_url":"https://github.com/apache/spark/pull/7753.diff","patch_url":"https://github.com/apache/spark/pull/7753.patch"},"body":"This is a sub-task of [SPARK-9103](https://issues.apache.org/jira/browse/SPARK-9103), we'd like to expose the memory usage for spark running time, this is the first step to expose the netty buffer used both with on-heap and off-heap memory. Also the metrics are showed on WebUI. In this PR, a new web Tab name `Memory` is added. Which is used to show the memory usage of each executors (can be in more details in future). the screenshot is like the following:\r\n\r\n![image](https://cloud.githubusercontent.com/assets/4716022/9379782/46e8d4a6-475e-11e5-8d39-f0556d569797.png)\r\n\r\nThis is for each stages memory info:\r\n![image](https://cloud.githubusercontent.com/assets/4716022/9379831/9a9a0250-475e-11e5-9d52-97f868605d11.png)\r\n\r\nThis is History View:\r\n![image](https://cloud.githubusercontent.com/assets/4716022/9379991/bd64f0e6-475f-11e5-8b12-15ea476195a7.png)\r\n\r\n\r\nThis is WIP because with unit tests left.\r\n  "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/153490332","html_url":"https://github.com/apache/spark/pull/7753#issuecomment-153490332","issue_url":"https://api.github.com/repos/apache/spark/issues/7753","id":153490332,"user":{"login":"squito","id":71240,"avatar_url":"https://avatars.githubusercontent.com/u/71240?v=3","gravatar_id":"","url":"https://api.github.com/users/squito","html_url":"https://github.com/squito","followers_url":"https://api.github.com/users/squito/followers","following_url":"https://api.github.com/users/squito/following{/other_user}","gists_url":"https://api.github.com/users/squito/gists{/gist_id}","starred_url":"https://api.github.com/users/squito/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squito/subscriptions","organizations_url":"https://api.github.com/users/squito/orgs","repos_url":"https://api.github.com/users/squito/repos","events_url":"https://api.github.com/users/squito/events{/privacy}","received_events_url":"https://api.github.com/users/squito/received_events","type":"User","site_admin":false},"created_at":"2015-11-03T21:12:20Z","updated_at":"2015-11-03T21:12:20Z","body":"Hi @liyezhang556520 thanks for updating.  Can you add some unit tests, especially MemoryListener?  Eg., push some events into an EventLoggingListener, then read those events back into the MemoryListener and make sure the stages have the right metrics.  You can use the cases you describe in your design doc.\r\n\r\nalso all the info available in the memory tab should also be available through the json api as well ... that can wait for a followup if you really need to."}}