{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5547","labels_url":"https://api.github.com/repos/apache/spark/issues/5547/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5547/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5547/events","html_url":"https://github.com/apache/spark/pull/5547","id":69033245,"number":5547,"title":"[SPARK-6418] Add simple per-stage visualization to the UI [WIP]","user":{"login":"shroffpradyumn","id":3763270,"avatar_url":"https://avatars.githubusercontent.com/u/3763270?v=3","gravatar_id":"","url":"https://api.github.com/users/shroffpradyumn","html_url":"https://github.com/shroffpradyumn","followers_url":"https://api.github.com/users/shroffpradyumn/followers","following_url":"https://api.github.com/users/shroffpradyumn/following{/other_user}","gists_url":"https://api.github.com/users/shroffpradyumn/gists{/gist_id}","starred_url":"https://api.github.com/users/shroffpradyumn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shroffpradyumn/subscriptions","organizations_url":"https://api.github.com/users/shroffpradyumn/orgs","repos_url":"https://api.github.com/users/shroffpradyumn/repos","events_url":"https://api.github.com/users/shroffpradyumn/events{/privacy}","received_events_url":"https://api.github.com/users/shroffpradyumn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-04-17T00:00:16Z","updated_at":"2015-04-20T22:45:15Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5547","html_url":"https://github.com/apache/spark/pull/5547","diff_url":"https://github.com/apache/spark/pull/5547.diff","patch_url":"https://github.com/apache/spark/pull/5547.patch"},"body":"Hello!\r\n\r\nI am working on adding a graph visualization to the Spark stage page (screenshots for 10, 100, and 1000 task graphs attached). Below are some details about my implementation:\r\n\r\nThe x-axis is the time axis (with the launch time of the first task as t = 0) and the y-axis is the task # (pretty self-explanatory). At present, I'm limiting the max number of tasks displayed on the graph to 1000 but I intend to add functionality to select the graph task-range during the second iteration.\r\n\r\nYou can hover over a specific part of the graph to see a tool-tip indicating the task #, phase, and the time taken by that phase (pretty useful when there are a lot of tasks on the graph). Also, the x-axis unit is milliseconds (if the longest task takes less than a second), seconds (if the longest task takes more than a second and less than a minute), and so on for minutes and hours.\r\n\r\nI also intend to add shuffle read and write times to the graph (once my midterms are over), and would love to get some suggestions, feedback, and criticism.\r\n\r\nSCREENSHOTS\r\n\r\n10 tasks:\r\n\r\n![10](https://cloud.githubusercontent.com/assets/3763270/7196476/677b710e-e487-11e4-8cfd-8727a913c8ea.png)\r\n\r\n100 tasks:\r\n![100](https://cloud.githubusercontent.com/assets/3763270/7196478/67a0cac6-e487-11e4-9aae-9502050b296f.png)\r\n\r\n1000 tasks:\r\n\r\n![1000](https://cloud.githubusercontent.com/assets/3763270/7196477/679ddcbc-e487-11e4-828c-74cbcb961a48.png)\r\n \r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/94585961","html_url":"https://github.com/apache/spark/pull/5547#issuecomment-94585961","issue_url":"https://api.github.com/repos/apache/spark/issues/5547","id":94585961,"user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"created_at":"2015-04-20T22:45:15Z","updated_at":"2015-04-20T22:45:15Z","body":"@pwendell talked about this and #2342 a little bit offline. Our feeling is that this is a more elegant representation of task times than #2342, especially when there are many tasks within a stage. One concern I have, however, what happens when you zoom (does it currently support zooming?). It would make little sense to zoom without keeping the axes, but my impression is that implementing this is pretty hard since we're directly using d3.\r\n\r\nBonus: It doesn't have to be part of this patch, but it would really cool if there's a mode where we can align the breakdown of the task times along the vertical axis. Right now you can't really compare the serialization time of the first task with that of the last task, let alone track whether it has grown incrementally over time. Realistically we will implement this separately say for 1.5, but I imagine this bonus feature is gonna be immensely useful."}}