{"action":"opened","issue":{"url":"https://api.github.com/repos/spotify/luigi/issues/1070","labels_url":"https://api.github.com/repos/spotify/luigi/issues/1070/labels{/name}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/1070/comments","events_url":"https://api.github.com/repos/spotify/luigi/issues/1070/events","html_url":"https://github.com/spotify/luigi/issues/1070","id":94735472,"number":1070,"title":"PySpark error - \"OSError: [Errno 2] No such file or directory\"","user":{"login":"dmcclory","id":1952789,"avatar_url":"https://avatars.githubusercontent.com/u/1952789?v=3","gravatar_id":"","url":"https://api.github.com/users/dmcclory","html_url":"https://github.com/dmcclory","followers_url":"https://api.github.com/users/dmcclory/followers","following_url":"https://api.github.com/users/dmcclory/following{/other_user}","gists_url":"https://api.github.com/users/dmcclory/gists{/gist_id}","starred_url":"https://api.github.com/users/dmcclory/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmcclory/subscriptions","organizations_url":"https://api.github.com/users/dmcclory/orgs","repos_url":"https://api.github.com/users/dmcclory/repos","events_url":"https://api.github.com/users/dmcclory/events{/privacy}","received_events_url":"https://api.github.com/users/dmcclory/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-13T14:33:35Z","updated_at":"2015-07-13T14:33:35Z","closed_at":null,"body":"I'm trying to run a PySparkTask from luigi, but I've been hitting this stack trace. I'm trying to run it locally for now:\r\n\r\n```\r\nINFO: [pid 45554] Worker Worker(salt=641770409, workers=1, host=dans-air, username=dan, pid=45554) running   PlanDesignBuilder(design_output=s3://picwell.vendors.promptworks/luigi_spike/trial_benefit_output.json)\r\nINFO: Running: ['spark-submit', '--deploy-mode', 'client', '--name', 'PlanDesignBuilder', '/Users/dan/prompt/picwell/etl/env/lib/python2.7/site-packages/luigi/contrib/pyspark_runner.py', '/var/folders/j5/ng5x9x4x4txgf82ykr7c_wlc0000gn/T/PlanDesignBuilderLZSP4w/PlanDesignBuilder.pickle']\r\nERROR: [pid 45554] Worker Worker(salt=641770409, workers=1, host=dans-air, username=dan, pid=45554) failed    PlanDesignBuilder(design_output=s3://picwell.vendors.promptworks/luigi_spike/trial_benefit_output.json)\r\nTraceback (most recent call last):\r\n  File \"/Users/dan/prompt/picwell/etl/env/lib/python2.7/site-packages/luigi/worker.py\", line 137, in run\r\n    new_deps = self._run_get_new_deps()\r\n  File \"/Users/dan/prompt/picwell/etl/env/lib/python2.7/site-packages/luigi/worker.py\", line 88, in _run_get_new_deps\r\n    task_gen = self.task.run()\r\n  File \"/Users/dan/prompt/picwell/etl/env/lib/python2.7/site-packages/luigi/contrib/spark.py\", line 336, in run\r\n    super(PySparkTask, self).run()\r\n  File \"/Users/dan/prompt/picwell/etl/env/lib/python2.7/site-packages/luigi/contrib/spark.py\", line 235, in run\r\n    universal_newlines=True)\r\n  File \"/Users/dan/.pyenv/versions/2.7.5/lib/python2.7/subprocess.py\", line 711, in __init__\r\n    errread, errwrite)\r\n  File \"/Users/dan/.pyenv/versions/2.7.5/lib/python2.7/subprocess.py\", line 1308, in _execute_child\r\n    raise child_exception\r\nOSError: [Errno 2] No such file or directory\r\n```\r\n\r\nluigi.conf:\r\n```\r\n[spark]\r\nspark-submit: /Users/dan/prompt/picwell/spark-1.4.0-bin-hadoop2.6/bin/spark-submit\r\nmaster: spark://localhost:7077\r\n```\r\n\r\nThis has been happening with luigi at 1.3.0, on spark 1.3 and 1.4.\r\n\r\nIssue #866 had a similar stack trace, but the proposed solution (making sure that the name parameter doesn't ) doesn't apply here.\r\n\r\nI'm happy to provide any other information! Does this seem like a bug or user error?"}}