{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5144","labels_url":"https://api.github.com/repos/apache/spark/issues/5144/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5144/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5144/events","html_url":"https://github.com/apache/spark/pull/5144","id":63842878,"number":5144,"title":"[SPARK-5338][MESOS] Add cluster mode support for Mesos","user":{"login":"tnachen","id":67244,"avatar_url":"https://avatars.githubusercontent.com/u/67244?v=3","gravatar_id":"","url":"https://api.github.com/users/tnachen","html_url":"https://github.com/tnachen","followers_url":"https://api.github.com/users/tnachen/followers","following_url":"https://api.github.com/users/tnachen/following{/other_user}","gists_url":"https://api.github.com/users/tnachen/gists{/gist_id}","starred_url":"https://api.github.com/users/tnachen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tnachen/subscriptions","organizations_url":"https://api.github.com/users/tnachen/orgs","repos_url":"https://api.github.com/users/tnachen/repos","events_url":"https://api.github.com/users/tnachen/events{/privacy}","received_events_url":"https://api.github.com/users/tnachen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-03-23T21:54:56Z","updated_at":"2015-03-25T22:50:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5144","html_url":"https://github.com/apache/spark/pull/5144","diff_url":"https://github.com/apache/spark/pull/5144.diff","patch_url":"https://github.com/apache/spark/pull/5144.patch"},"body":"This patch adds the support for cluster mode to run on Mesos.\r\nIt introduces a new Mesos framework dedicated to launch new apps/drivers, and can be called with the spark-submit script and specifying --master flag to the cluster mode REST interface instead of Mesos master.\r\n\r\nExample:\r\n./bin/spark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi --master mesos://10.0.0.206:8077 --executor-memory 1G --total-executor-cores 100 examples/target/spark-examples_2.10-1.3.0-SNAPSHOT.jar 30\r\n\r\nPart of this patch is also to abstract the StandaloneRestServer so it can have different implementations of the REST endpoints.\r\n\r\nFeatures of the cluster mode in this PR:\r\n- Supports supervise mode where scheduler will keep trying to reschedule exited job.\r\n- Adds a new UI for the cluster mode scheduler to see all the running jobs, finished jobs, and supervise jobs waiting to be retried\r\n- Supports state persistence to ZK, so when the cluster scheduler fails over it can pick up all the queued and running jobs\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/86243643","html_url":"https://github.com/apache/spark/pull/5144#issuecomment-86243643","issue_url":"https://api.github.com/repos/apache/spark/issues/5144","id":86243643,"user":{"login":"andrewor14","id":2133137,"avatar_url":"https://avatars.githubusercontent.com/u/2133137?v=3","gravatar_id":"","url":"https://api.github.com/users/andrewor14","html_url":"https://github.com/andrewor14","followers_url":"https://api.github.com/users/andrewor14/followers","following_url":"https://api.github.com/users/andrewor14/following{/other_user}","gists_url":"https://api.github.com/users/andrewor14/gists{/gist_id}","starred_url":"https://api.github.com/users/andrewor14/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andrewor14/subscriptions","organizations_url":"https://api.github.com/users/andrewor14/orgs","repos_url":"https://api.github.com/users/andrewor14/repos","events_url":"https://api.github.com/users/andrewor14/events{/privacy}","received_events_url":"https://api.github.com/users/andrewor14/received_events","type":"User","site_admin":false},"created_at":"2015-03-25T22:50:58Z","updated_at":"2015-03-25T22:50:58Z","body":"Hi @tnachen thanks a lot for working on this. Mesos is the last mode that does not support cluster mode and it would wonderful to see it reach parity on this regard.\r\n\r\nOverall I think the high level approach is reasonable, but there are a few concerns I had about the code organization and style. In particular, there is a lot of code to be reused between the Mesos and standalone submission gateways, and you have refactored this nicely by making `RestServer` an abstract class. However, there are many instances where the Mesos code just reaches into the standalone packages to grab specific values or classes from there. A cleaner way to refactor this is to abstract all common fields and classes that need to be reused in common packages (even `o.a.s.deploy.` is probably fine).\r\n\r\nThe other thing is that this patch creates many small classes, most of which I believe are unnecessary. Having many related small classes imposes readability burden and makes the code much harder to follow. For instance, we probably shouldn't follow YARN's favorite pattern of making an abstract parent class that only has one implementation (e.g. `ContainerManagerImpl`) for `MesosClusterScheduler`. I pointed out other examples inline (e.g. `MesosDriverDescription`). IIUC the motivation behind some of these classes is to follow closely the standalone mode behavior, but this is actually not a requirement. In the worst case, this could lead to maintenance burden in the future if we tie the implementation of the two modes together excessively.\r\n\r\nLastly, there are a lot of code style deviations from the rest of the code base. Good references include the [Spark style guide](https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide) or the [Databricks style guide](https://github.com/databricks/scala-style-guide), which was written around writing code in Spark."}}