{"action":"created","issue":{"url":"https://api.github.com/repos/gorillalabs/sparkling/issues/2","labels_url":"https://api.github.com/repos/gorillalabs/sparkling/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/gorillalabs/sparkling/issues/2/comments","events_url":"https://api.github.com/repos/gorillalabs/sparkling/issues/2/events","html_url":"https://github.com/gorillalabs/sparkling/issues/2","id":53767325,"number":2,"title":"Initial job has not accepted any resources – Clojure-specific Problem","user":{"login":"rizo","id":308413,"avatar_url":"https://avatars.githubusercontent.com/u/308413?v=3","gravatar_id":"","url":"https://api.github.com/users/rizo","html_url":"https://github.com/rizo","followers_url":"https://api.github.com/users/rizo/followers","following_url":"https://api.github.com/users/rizo/following{/other_user}","gists_url":"https://api.github.com/users/rizo/gists{/gist_id}","starred_url":"https://api.github.com/users/rizo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rizo/subscriptions","organizations_url":"https://api.github.com/users/rizo/orgs","repos_url":"https://api.github.com/users/rizo/repos","events_url":"https://api.github.com/users/rizo/events{/privacy}","received_events_url":"https://api.github.com/users/rizo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-01-08T16:09:50Z","updated_at":"2015-01-09T21:06:08Z","closed_at":null,"body":"Hi,\r\n\r\nI have a standalone spark cluster running on my test machine (version 1.1.0 compiled from source without hadoop) and want to use it in a simple demo app with sparkling.\r\n\r\nHere's a minimal source code that I used to reproduce the problem:\r\n\r\n- <https://github.com/rizo/spark-playground/tree/master/spark-base/src/spark_base>\r\n\r\nThere are no other workers on the cluster using the memory. Here is the status of the cluster before the execution of the job:\r\n\r\n![screen shot 2015-01-08 at 15 20 54](https://cloud.githubusercontent.com/assets/308413/5665306/739944a0-974f-11e4-8d42-624df0933683.png)\r\n\r\nWhen I run the demo app, the following error is risen:\r\n\r\n```\r\nInitial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory.\r\n```\r\n\r\n(Full job logs: <https://gist.github.com/rizo/2662a1a4c81d1c572d09>)\r\n\r\nAfter the submission, the application occupies all of the cores and uses 512mb and exits after 30 sec timeout:\r\n\r\n![screen shot 2015-01-08 at 15 21 17](https://cloud.githubusercontent.com/assets/308413/5665376/e1a035f8-974f-11e4-86b6-aa7e2e7b0c83.png)\r\n\r\nInitially I thought there was a problem with my cluster configuration, but spark-shell and pyspark are working without any problems. Also I successfully executed the job with spark-submit using the uberjar built with the same spark-base project.\r\n\r\nWhat could be wrong here? Are there anything clojure-specific settings I have to set to run the app on the cluster?\r\n\r\nThanks in advance for your response!"},"comment":{"url":"https://api.github.com/repos/gorillalabs/sparkling/issues/comments/69398873","html_url":"https://github.com/gorillalabs/sparkling/issues/2#issuecomment-69398873","issue_url":"https://api.github.com/repos/gorillalabs/sparkling/issues/2","id":69398873,"user":{"login":"rizo","id":308413,"avatar_url":"https://avatars.githubusercontent.com/u/308413?v=3","gravatar_id":"","url":"https://api.github.com/users/rizo","html_url":"https://github.com/rizo","followers_url":"https://api.github.com/users/rizo/followers","following_url":"https://api.github.com/users/rizo/following{/other_user}","gists_url":"https://api.github.com/users/rizo/gists{/gist_id}","starred_url":"https://api.github.com/users/rizo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rizo/subscriptions","organizations_url":"https://api.github.com/users/rizo/orgs","repos_url":"https://api.github.com/users/rizo/repos","events_url":"https://api.github.com/users/rizo/events{/privacy}","received_events_url":"https://api.github.com/users/rizo/received_events","type":"User","site_admin":false},"created_at":"2015-01-09T21:06:08Z","updated_at":"2015-01-09T21:06:08Z","body":"I think I misunderstood how the driver should be launched. I tried to directly execute my code on the cluster's machine by invoking `lein repl` and providing the Spark's master URI.\r\n\r\nI was convinced that it should work because I did exactly that in the shell (both in Scala and Python variants) by manually creating the conf and context and it worked. I did the same in Clojure's repl and was always getting the resources' problem I described initially.\r\n\r\nBy submitting the application with your instructions I was able to run the code and see the expected result. I still don't understand what was wrong with my initial approach and IMO the documentation is not very clear on this particular aspect. Also I found Spark's error messages (which appeared nondeterministically several times) hard to understand and debug, which made it difficult to locate the source of the problem.\r\n\r\nIf you don't have any further comments you may close the issue.\r\nAnd thank you for your help! ;)"}}